CN115469811A

CN115469811A - Data storage method, apparatus, device, storage medium and program product

Info

Publication number: CN115469811A
Application number: CN202211162016.3A
Authority: CN
Inventors: 黎兴建; 廖志军; 徐畅
Original assignee: Guangzhou Ankai Microelectronics Co ltd
Current assignee: Guangzhou Ankai Microelectronics Co ltd
Priority date: 2022-09-23
Filing date: 2022-09-23
Publication date: 2022-12-13

Abstract

The application relates to a data storage method, apparatus, device, storage medium and program product. The method comprises the following steps: for a first target file to be stored in the ROMFS file system, determining whether compression processing needs to be executed on the first target file according to the type of the first target file and the file header of the first target file; the first target file comprises the file header and file data; under the condition that the first target file needs to be compressed, determining a compression strategy according to the residual storage space in the memory, and compressing the file data based on the compression strategy to obtain compressed data corresponding to the first target file; and writing the file header and the compressed data into a target image file, and storing the target image file into the memory. By adopting the method, the storage cost can be reduced and more data can be stored.

Description

Data storage method, apparatus, device, storage medium and program product

Technical Field

The present application relates to the field of data processing technologies, and in particular, to a data storage method, apparatus, device, storage medium, and program product.

Background

Nowadays, storage devices are widely used in various electronic products, and with the continuous improvement and development of functions of various electronic products, data required to be stored by various electronic products is increasing, and thus, a required Flash storage space is also increasing. If the data storage of electronic products is to be supported, the Flash storage space needs to be increased. However, increasing the Flash storage space will result in increased costs. Therefore, there is a need to solve the problem of storing more data without increasing the cost.

Disclosure of Invention

In view of the above, it is necessary to provide a data storage method, an apparatus, a device, a storage medium, and a program product capable of reducing storage cost and storing more data in response to the above technical problems.

In a first aspect, the present application provides a data storage method. The method comprises the following steps:

for a first target file to be stored in the ROMFS file system, determining whether compression processing needs to be executed on the first target file according to the type of the first target file and the file header of the first target file; the first target file comprises the file header and file data;

under the condition that the first target file needs to be compressed, determining a compression strategy according to the residual storage space in the memory, and compressing the file data based on the compression strategy to obtain compressed data corresponding to the first target file;

and writing the file header and the compressed data into a target image file, and storing the target image file into the memory.

In one embodiment, the determining whether the compression process needs to be performed on the first target file according to the type of the first target file and the file header of the first target file includes: if the type of the first target file is a target type and the target bit in the file type parameter is a preset bit value, determining that the compression processing needs to be executed on the first target file; the target type is a type that file data in the file is not empty.

In one embodiment, the target bit is a 0 th bit to a 2 nd bit in the file type parameter, and the preset bit value includes a 0 th bit value of 0, a 1 st bit value of 1 and a 2 nd bit value of 1.

In one embodiment, the compressing the file data based on the compression policy to obtain compressed data corresponding to the first target file includes: compressing the file data based on the compression strategy to obtain a compressed file, and generating a compression header, wherein the compression header comprises compression information related to compression; and taking the compressed file and the compression header as the compressed data.

In one embodiment, the compressing policy includes a compressed data amount, and the compressing the file data based on the compressing policy to obtain a compressed file includes: extracting a plurality of data blocks from the file data according to the compressed data amount; and respectively compressing each data block to obtain the compressed file.

In one embodiment, the header includes a file value, the file value being used to characterize the size of the first target file; the generating a compression head, comprising: obtaining a bitmap table; the bitmap table comprises bitmap information corresponding to each data block, and the bitmap information is used for representing the storage position of the data block; wherein, the bitmap information is generated in the process of writing the compressed data into the target image file; acquiring the file value according to the file header; and taking the bitmap table, the compressed data amount and the file value as the compression information.

In one embodiment, before writing the header and the compressed data into the target image file, the method further comprises: determining a compression value of the compressed data, the compression value being indicative of a size of the compressed data; taking the compressed value as the file value; correspondingly, the writing the file header and the compressed data into the target image file includes: and writing the file header containing the compression value and the compression data into the target image file.

In one embodiment, the storing the compressed data in a temporary folder, and the writing the header and the compressed data into a target image file, includes: creating a target mirror image file; and reading data blocks from the temporary folder according to the compressed data amount, and writing the read data blocks into the target image file in sequence.

In one embodiment, the method further comprises: receiving a file acquisition request sent by an upper application, and determining a target file header corresponding to a second target file according to the file acquisition request; determining whether the second target file executes compression processing according to the target file header; if the second target file is determined to execute the compression processing, reading a target compression head corresponding to the second target file; and acquiring target file data corresponding to the second target file with a preset data length according to the target compression head, wherein the preset data length is determined according to the file acquisition request.

In one embodiment, the obtaining target file data corresponding to the second target file with a preset data length according to the target compression header includes: determining a target storage position of each data block in the target compressed data according to a bitmap table in the target compression head, and determining a target compressed data volume from the target compression head; reading compressed data with preset data length from the target storage position according to the target compressed data amount to obtain one or more target data blocks; and obtaining the target file data according to the target data block.

In one embodiment, the obtaining the target file data according to the target data block includes: determining whether the target data block needs to be decompressed according to the byte parameters of the target data block; and if the target data block is determined to need to be decompressed, decompressing the target data block to obtain the target file data.

In a second aspect, the present application further provides a data storage device. The device comprises:

the device comprises a first determining module, a second determining module and a compressing module, wherein the first determining module is used for determining whether compression processing needs to be executed on a first target file to be stored in the ROMFS file system according to the type of the first target file and the file header of the first target file; the first target file comprises the file header and file data;

the compression module is used for determining a compression strategy according to the residual storage space in the storage under the condition of determining that the compression processing needs to be executed on the first target file, and compressing the file data based on the compression strategy to obtain compressed data corresponding to the first target file;

and the storage module is used for writing the file header and the compressed data into a target image file and storing the target image file into the memory.

In one embodiment, the file header includes a file type parameter characterized by a plurality of bits, and the first determining module is specifically configured to: if the type of the first target file is a target type and the target bit in the file type parameter is a preset bit value, determining that the compression processing needs to be executed on the first target file; wherein the target type is a type that file data in the file is not empty.

In one embodiment, the target bits are 0 th bit to 2 nd bit in the file type parameter, and the preset bit value includes a 0 th bit value of 0, a 1 st bit value of 1, and a 2 nd bit value of 1.

In one embodiment, the compression module is specifically configured to: compressing the file data based on the compression strategy to obtain a compressed file, and generating a compression head, wherein the compression head comprises compression information related to compression; and taking the compressed file and the compression header as the compressed data.

In one embodiment, the compression module is specifically configured to: extracting a plurality of data blocks from the file data according to the compressed data amount; and respectively compressing each data block to obtain the compressed file.

In one embodiment, the header includes a file value, the file value being used to characterize the size of the first target file; the compression module is specifically configured to: obtaining a bitmap table; the bitmap table comprises bitmap information corresponding to each data block, and the bitmap information is used for representing the storage position of the data block; wherein, the bitmap information is generated in the process of writing the compressed data into the target image file; acquiring the file value according to the file header; and taking the bitmap table, the compressed data amount and the file value as the compression information.

In one embodiment, before writing the header and the compressed data into the target image file, the apparatus further includes:

a second determining module, configured to determine a compression value of the compressed data, where the compression value is used to characterize a size of the compressed data; taking the compressed value as the file value;

correspondingly, the storage module is specifically configured to: and writing the file header containing the compressed value and the compressed data into the target image file.

In one embodiment, the compressed data is stored in a temporary folder, and the storage module is specifically configured to: creating a target mirror image file; and reading data blocks from the temporary folder according to the compressed data amount, and writing the read data blocks into the target image file in sequence.

In one embodiment, the apparatus further comprises:

the receiving module is used for receiving a file acquisition request sent by an upper application and determining a target file header corresponding to a second target file according to the file acquisition request;

the acquisition module is used for determining whether the second target file executes compression processing or not according to the target file header; if the second target file is determined to execute the compression processing, reading a target compression head corresponding to the second target file; and acquiring target file data corresponding to the second target file with a preset data length according to the target compression head, wherein the preset data length is determined according to the file acquisition request.

In one embodiment, the obtaining module is specifically configured to: determining the target storage position of each data block in the target compressed data according to the bitmap in the target compression head, and determining the target compressed data volume from the target compression head; reading compressed data with preset data length from the target storage position according to the target compressed data amount to obtain one or more target data blocks; and obtaining the target file data according to the target data block.

In one embodiment, the obtaining module is specifically configured to: determining whether the target data block needs to be decompressed according to the byte parameters of the target data block; and if the target data block is determined to need to be decompressed, decompressing the target data block to obtain the target file data.

In a third aspect, the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method according to any one of the above first aspects when executing the computer program.

In a fourth aspect, the present application further provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the method of any one of the above first aspects.

In a fifth aspect, the present application also provides a computer program product comprising a computer program which, when executed by a processor, performs the steps of the method of any one of the above first aspects.

According to the data storage method, the data storage device, the data storage equipment, the data storage medium and the program product, for a first target file to be stored in the ROMFS file system and comprising a file header and file data, whether compression processing needs to be executed on the first target file is determined according to the type of the first target file and the file header of the first target file; under the condition that the first target file needs to be compressed, determining a compression strategy according to the residual storage space in the memory, and compressing file data based on the compression strategy to obtain compressed data corresponding to the first target file; furthermore, the file header and the compressed data can be written into the target image file, and the target image file is stored into the memory. Thus, for the device adopting the ROMFS file system, when the mirror image is generated through the ROMFS file system, the first target file which can be compressed is compressed, and then the first target file is written into the mirror image file and stored into the memory, so that the storage space of the memory is greatly saved, and the device can store more data under the condition of not increasing the cost.

Drawings

FIG. 1 is a schematic flow chart diagram illustrating a data storage method according to an embodiment;

FIG. 2 is a diagram of a file system structure in one embodiment;

FIG. 3 is a diagram of a file structure in one embodiment;

FIG. 4 is a schematic flow chart illustrating the determination of compressed data in one embodiment;

FIG. 5 is a flow diagram illustrating obtaining a compressed file according to one embodiment;

FIG. 6 is a flow diagram illustrating the determination of a compression header in one embodiment;

FIG. 7 is a diagram illustrating a compressed file data structure, according to an embodiment;

FIG. 8 is a flow diagram illustrating the writing of compressed data in one embodiment;

FIG. 9 is a schematic flow chart diagram illustrating another method for storing data in one embodiment;

FIG. 10 is a schematic flow chart illustrating the determination of target document data in one embodiment;

FIG. 11 is a schematic logic flow diagram illustrating the file storage process in one embodiment;

FIG. 12 is a schematic logic flow diagram illustrating the file read process in one embodiment;

FIG. 13 is a block diagram of a data storage device in one embodiment;

FIG. 14 is a diagram illustrating an internal structure of a computer device according to an embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of and not restrictive on the broad application.

At present, storage devices are widely applied to various electronic devices, and the processing requirements of the electronic devices are increasing, for example, stored data is more, and supported functions are increasing, taking the electronic device as a camera as an example, the requirements on the resolution of the camera are increasing nowadays, correspondingly, details of image processing are becoming finer and finer, parameters of image processing are increasing, and image parameter files configured to the electronic devices are increasing, and thus, required storage space is increasing. Files required to be stored by the electronic device are typically stored in a memory, such as a FLASH memory. If the number of the storage files increases, the storage space of the memory needs to be increased. However, if the FLASH memory space is large, the cost is relatively increased, which results in a decrease in the competitiveness of the electronic device. Therefore, there is a need to solve the problem of enabling electronic devices to store more data files without increasing or reducing the cost.

In view of this, embodiments of the present application provide a data storage method, which enables an electronic device to store more data while reducing storage cost.

In an embodiment, as shown in fig. 1, a data storage method is provided, and the embodiment of the present application is described by taking an example that the method is applied to a terminal. The ROMFS file system is deployed in the terminal, the terminal can be but not limited to various personal computers, notebook computers, smart phones, tablet computers, internet of things equipment, camera equipment and portable wearable equipment, and the Internet of things equipment can be smart sound boxes, smart televisions, smart air conditioners, smart vehicle-mounted equipment and the like. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device and the like, and the camera device can be an electronic device and the like which comprises a camera and can store or process images directly shot by the camera. Optionally, the method may also be applied to a server, and the server may be an individual server or a server cluster. The data storage method comprises the following steps:

step 101, for a first object file to be stored in the ROMFS file system, determining whether compression processing needs to be performed on the first object file according to the type of the first object file and a file header of the first object file; the first object file includes a file header and file data.

The ROMFS file system is a read-only file system, is relatively simple, occupies a small space, and has the characteristics of small volume, good reliability, high reading speed, and the like. The terminal can be deployed with an ROMFS file system, and data files transmitted by upper-layer applications in the terminal are stored in the memory through the ROMFS file system. Accordingly, the data file stored in the memory may also be read by the ROMFS file system. The memory refers to a flash memory, for example.

The first target file refers to a file which is acquired by the terminal and needs to be stored or a file which is already stored in the terminal, and the first target file can be programmed into a memory for storage through the ROMFS file system. The structure of a file stored in the ROMFS file system is generally composed of a file header and file data, wherein the file header specifically includes basic information of the file, and the file data is valid data actually contained in the file. The first target file may be a folder or a file, and if the first target file is a file, file data specifically included in the file may be, for example, data such as text, pictures, video, and audio, which is not specifically limited in this embodiment of the application.

In the embodiment of the application, whether the first target file can be compressed or not can be determined based on the type of the first target file and the file header of the first target file, so that the first target file can be compressed and stored under the condition that the first target file can be compressed, and the occupied storage space is reduced. The type of the first target file is determined whether the first target file is a file containing valid data or a file not containing valid data. The header information may be predetermined by the upper application, and thus, information indicating whether the first object file can be compressed is included in the header.

And if the type of the first target file is judged to be the type containing the effective data and the information in the file header indicates that the file can be compressed, determining that the first target file needs to be compressed.

And 102, under the condition that the compression processing needs to be executed on the first target file, determining a compression strategy according to the residual storage space in the memory, and compressing the file data based on the compression strategy to obtain compressed data corresponding to the first target file.

If it is determined that the compression processing can be performed on the first target file, the terminal first needs to determine a compression policy based on the storage space remaining in the memory, and perform compression processing based on the determined compression policy. Based on the method, different compression strategies can be determined according to the size of the storage space, and for the different compression strategies, the storage space occupied by compressed data obtained after compression and the compression efficiency are different, so that the flexibility of compression processing can be effectively improved, and the effectiveness of utilization of the storage space of the memory is improved. Optionally, the compression policy includes compressing the file data according to a preset block size.

And 103, writing the file header and the compressed data into the target image file, and storing the target image file into a memory.

In general, the ROMFS file system writes a first target file into an image file, and writes the image file into memory to implement storage of the first target file. The embodiment of the application is based on the process, and compression processing is added in the process. Correspondingly, after the file data of the first target file is compressed, the obtained compressed data is used for replacing the file data and storing the file data into the memory, so that the file header and the compressed data of the first target file can be used as the processed first target file and written into the image file and further stored into the memory, and the first target file is compressed and then stored. Because the size of the compressed data is smaller than that of the file data, the storage space occupied after the compression processing is reduced, and the storage space is saved.

Optionally, a compression tool may be deployed in the ROMFS file system, where the compression tool is generated by using a software algorithm, and the terminal uses the compression tool to compress the file data according to a compression policy.

According to the data storage method, for a first target file to be stored in the ROMFS file system and comprising a file header and file data, whether compression processing needs to be executed on the first target file is determined according to the type of the first target file and the file header of the first target file; under the condition that the first target file needs to be compressed, determining a compression strategy according to the residual storage space in the memory, and compressing file data based on the compression strategy to obtain compressed data corresponding to the first target file; furthermore, the file header and the compressed data can be written into the target image file, and the target image file is stored in the memory. Thus, for the device adopting the ROMFS file system, when the mirror image is generated through the ROMFS file system, the first target file which can be compressed is compressed, and then the first target file is written into the mirror image file and stored into the memory, so that the storage space of the memory is greatly saved, and the device can store more data under the condition of not increasing the cost.

In one embodiment, the file header includes a file type parameter characterized by a plurality of bits, and determining whether a compression process needs to be performed on the first object file according to the type of the first object file and the file header of the first object file includes: if the type of the first target file is a target type and the target bit in the file type parameter is a preset bit value, determining that compression processing needs to be executed on the first target file; the target type is a type that file data in the file is not empty.

For ease of understanding, the partition structure of the ROMFS file system will be described first. Specifically, please refer to fig. 2, which illustrates a schematic structural diagram of a file system according to an embodiment of the present application. As can be seen, the ROMFS file system is entirely divided into two parts, a file system header area and a file data area. The header area of the file system includes basic information of the file system, specifically, the magic word is an identifier of the ROMFS file system, for example, "-rom1fs-" which may represent information such as a version of the file system; full size refers to the size of the ROMFS file system, in bytes; checksum, volume name refers to the name of the file system. The file data area files includes a plurality of specific files, each of which is composed of a file header and a file data, wherein the file header refers to a file header of the file, and the file data refers to file data of the file.

Specifically, please refer to fig. 3, which illustrates a file structure diagram provided in an embodiment of the present application, where a header file header includes basic information of a file, and file data is file data. Specifically, the next file hdr is the file type parameter, the file type parameter is represented by a plurality of bits, specifically, the high 28 bits indicate an offset address of the next file, the third bit is used to indicate whether the file can be executed, and the 2 nd bit to the 0 th bit are used to indicate the data type of the file. Info is related to a specific data type of a file, different data types, the meaning of the field is different. size characterizes the size of the file. checksum is the checksum. The file name is a file name, is not fixed in length, and is typically a 16-byte aligned string. For the ROMFS file system, each bit of the file type parameter takes a value of 0 or 1, and for the 2 nd bit to the 0 th bit, there may be 8 combinations, so that the supported file data types are 8 and cannot be expanded. The applicant finds that, based on practical application, there is an unused combination case in 8 combinations of bits from 2 nd bit to 0 th bit, and therefore, the type of the combination can be used for the compression type in the embodiment of the present application, so that in the case that the target bit in the file type parameter of the first target file is a preset bit value, that is, the combination is determined to be a file of a compressible type, and then the compression process can be performed on the file.

On this basis, as mentioned above, the type of the first target file may be a folder or a file, wherein the file data in the folder is empty and the file data in the file is not empty. If the first target file is determined to be a file, and the file data is not empty, the first target file is determined to be subjected to compression processing.

A combination of bits that can be used as a compression type will be explained below.

Specifically, the combination of 8 file types supported by the ROMFS file system, bit 0 to bit 2 being 110, is not used, that is, other combinations are usually used to characterize the file as different types of files. Therefore, in the embodiment of the present application, if the bits 0 to 2 in the file type parameter of the file header of the first target file transmitted by the upper layer application are 110, it is determined that the first target file is a compression type capable of performing compression processing, and if it is further determined that the first target file is also a target type whose file data is not empty, the first target file may be directly compressed, thereby saving a storage space.

Optionally, the format of the compression process may be named RFNEXT _ LZ4, and of course, other named identifiers may also be used, which is not specifically limited in this embodiment of the present application.

Based on this, the embodiment of the application is fully based on the original file structure of the ROMFS file system, and flexibly determines the compression type by using the file type supported by the ROMFS file system, so that the terminal performs compression processing on the first target file which can be compressed by using the ROMFS file system, thereby saving the storage space and reducing the storage cost.

The process of determining the compressed data will be explained below.

Please refer to fig. 4, which illustrates a flowchart of determining compressed data according to an embodiment of the present application. Compressing the file data based on the compression strategy to obtain compressed data corresponding to the first target file, including:

step 401, performing compression processing on file data based on a compression policy to obtain a compressed file, and generating a compression header, where the compression header includes compression information related to the compression processing.

Step 402, using the compressed file and the compressed header as compressed data.

As described above, in general, the structure of a file in the ROMFS file system is composed of a header and file data, and the file data includes valid data in the file. Data obtained after the file data is compressed is collectively called compressed data.

In the embodiment of the application, the file data area of the first target file which can be compressed is redesigned, so that the first target file comprises a compression header and a compressed file instead of the original file data after the compression processing is performed.

Wherein, the compression header comprises a plurality of compression information related to the compression processing. Specific compression information is described below.

In one embodiment, the compression policy includes compressing an amount of data. Please refer to fig. 5, which illustrates a flowchart of acquiring a compressed file according to an embodiment of the present application. Compressing the file data based on the compression strategy to obtain a compressed file, comprising:

step 501, extracting a plurality of data blocks from file data according to the amount of compressed data.

Step 502, performing compression processing on each data block respectively to obtain a compressed file.

The compression strategy is to compress the file data according to blocks (blocks), that is, to compress the file data of the first target file in an Lz4 format by taking the blocks as a unit. The amount of compressed data refers to the size of the block. For example, if it is determined from the remaining storage space that the amount of compressed data included in the compression policy is 4k, the file data is extracted in units of 4k, and a plurality of data blocks are obtained. And further compressing each data block according to an Lz4 format to obtain a plurality of compressed blocks, and forming a compressed file.

Alternatively, the compressed data amount may be 4k, 64k, or other different byte sizes. The process of determining the compression policy from the remaining storage space may include: if the residual storage space is larger than a preset space threshold value, a compression strategy is obtained by taking the first byte as a compression data volume; and if the residual storage space is smaller than the preset space threshold, obtaining a compression strategy by taking a second byte as a compressed data volume, wherein the first byte is 64k, and the second byte is 4k. It should be noted that the first byte and the second byte may also be other values, and may be determined according to the computing power of the terminal and the remaining storage space, which is not limited herein.

The first byte is used for extracting the data block for compression, so that the compression efficiency is high; the data block is extracted by the second byte for compression processing, and the occupied storage space is small, so that the flexibility of compression processing based on a compression strategy is high.

Optionally, after the plurality of data blocks are extracted, based on a storage requirement of an upper layer application, the terminal may compress some data blocks, not compress some data blocks, and use the compressed blocks obtained by the compression process and the data blocks that are not compressed as compressed files. In other words, the process of partial compression processing of the first target file is realized.

In the embodiment of the application, the file data extraction data blocks are compressed based on the compressed data volume, and the compressed data volume is different under the condition that the rest storage space is different, so that different compression processing can be flexibly executed based on the storage condition of a memory and the calculation efficiency condition of a terminal, the compression efficiency is improved, the storage occupancy rate is reduced, and the resources are saved.

Specific information contained in the compression header will be described below.

Please refer to fig. 6, which illustrates a schematic flow chart of determining a compression header according to an embodiment of the present application. The file header of the first target file comprises a file value, and the file value is used for representing the size of the first target file; the process of generating a compression head, comprising:

step 601, a bitmap table is obtained. The bitmap table comprises bitmap information corresponding to each data block, and the bitmap information is used for representing the storage position of the data block; wherein, the bitmap information is generated in the process of writing the compressed data into the target image file.

Step 602, obtaining a file value according to a file header.

Step 603, the bitmap table, the compressed data amount, and the file value are used as compressed information.

The structure of the compression header includes information areas such as the amount of compressed data, bit map table, original size of file, etc. Wherein the amount of compressed data is determined based on the remaining storage space, and the compression header may be written to during the compression process. The file value refers to the size in the file header, and when the first target file is obtained, the size of the file value can be obtained from the file header, and the file value can be written into the compression header.

The bitmap table area includes bitmap information of storage locations where each data block is compressed and stored. The bitmap information is, for example, an offset address of each data block for quickly indexing to the location of the data block in memory.

Specifically, as described above, the data blocks are extracted from the compressed data and compressed to obtain a compressed file, and the compressed file is composed of a plurality of compressed blocks. And in the process of writing the compressed data into the mirror image file, the terminal simultaneously records the bitmap information of each compressed block, and writes the bitmap information of each compressed block into the bitmap table area to form the bitmap table. When the upper layer application needs to extract each data block, the position of the corresponding compression block can be determined based on the bitmap table, so that the corresponding compression block is extracted and decompressed to obtain the required data block.

Please refer to fig. 7, which illustrates a schematic diagram of a compressed file data structure according to an embodiment of the present application. The file data is compressed data, and specifically comprises a compression header and a compression file, wherein the compression file comprises a plurality of compressed data blocks, and each compression block is obtained by compressing a corresponding data block. The Lz4 info in the compression header comprises the amount of compressed data and other data processed by compression; the file actual size in the compression header is a file value, which can be obtained from the file header; the compression header may also include a checksum; the data map in the compression header is a bitmap, wherein the offset of each compression block is recorded, so that the position of the compression block in the flash can be quickly indexed when the data map is read conveniently.

In the embodiment of the application, the file data area is redesigned, and the compressed data after the compression processing further comprises a file header, which can be based on the specific position of the compression block and the compression information related to the compression process, so that the index is convenient to read.

In one embodiment, before writing the header and the compressed data into the target image file, the method further comprises: determining a compression value of the compressed data, wherein the compression value is used for representing the size of the compressed data; taking the compressed value as a file value; correspondingly, writing the file header and the compressed data into the target image file comprises the following steps: and writing the file header containing the compressed value and the compressed data into the target image file.

For example, the file header in fig. 3 includes a file value size, which represents the size of the first target file. After the file data is compressed, the size of the first target file is reduced, and the size value is changed. After the compression process, the terminal may determine a specific size, that is, a compression value, of the compressed first target file according to the compressed data. At this time, the compression value represents the current specific size of the first target file instead of the file value. Accordingly, the original size of the first target file, i.e., the file value, is stored in the compression header. Therefore, for the first target file after the compression processing, the file value in the first target file is represented by the compression value, the current size of the first target file can be accurately described, and confusion is avoided.

In one embodiment, as shown in fig. 8, a schematic flow chart of writing compressed data provided by an embodiment of the present application is shown. Wherein, the compressed data is stored in the temporary folder, and the file header and the compressed data are written into the target mirror image file, including:

step 801, create a target image file.

And step 802, reading data blocks from the temporary folder according to the compressed data amount, and writing the read data blocks into the target image file in sequence.

After the terminal reads the first target file through the ROMFS file system, after the first target file is judged to be of the target type and can be compressed, the file data of the first target file are compressed, and the compressed data after the compression processing are stored in the temporary folder.

The ROMFS file system is provided with an image generation tool, and a terminal creates a target image file through the image generation tool. In the process of writing the compressed data into the target image file, specifically, the data blocks are read from the temporary folder in sequence through the amount of the compressed data recorded in the compression header, so as to write each data block into the target image file specifically, in the process of writing the data blocks, the bitmap information of each data block is generated simultaneously, and at this time, the read data block is the compression block.

The above description is directed to the storage process in which the first target file is a file whose file data is not empty. In one case, the first target file to be stored may also be a folder, and the folder further includes a file or a folder, which aims to compress data of the file in the folder. In the process, if the first target file is judged to be a folder, recording a file header of the folder into a linked list, and entering the folder to read a file of a next layer; if the next layer is read as a file, compressing the file under the condition that the file can be compressed, storing a compressed block obtained by compression in a temporary folder and recording a file header of the file into a linked list; and continuing to read the next file for corresponding processing. For the traversed files or folders, the file header information is recorded in a linked list. After the target image file is created, the information in the linked list is sequentially read, and the corresponding file or folder is written into the target image file according to the information in the linked list so as to complete the manufacture of the image file.

The compression processing in the data storage process is explained above. Accordingly, the above data storage method naturally further includes a process of decompressing the compressed data, and the process of decompressing will be described below.

In an embodiment, please refer to fig. 9, which illustrates a flowchart of another data storage method provided in an embodiment of the present application. The data storage method further comprises:

and 104, receiving a file acquisition request sent by the upper application, and determining a target file header corresponding to the second target file according to the file acquisition request.

When the upper layer application in the terminal needs to acquire the file data corresponding to the second target file, the upper layer application may send a file acquisition request to the ROMFS file system. And determining a target file header corresponding to the second target file according to the file acquisition request, and further acquiring a target data file based on information in the target file header.

And step 105, determining whether the second target file executes compression processing according to the target file header.

And if so, the second target file is stored in a memory after being compressed, and at the moment, the second target file is determined to be subjected to compression processing.

And step 106, reading a target compression header corresponding to the second target file if the second target file is determined to be subjected to the compression processing.

If it is determined that the second target file performs the compression processing, and accordingly, the corresponding compression block needs to be read based on information in a target compression header corresponding to the second target file, so as to decompress the compression block to obtain the target data file.

And step 107, acquiring target file data corresponding to a second target file with a preset data length according to the target compression header. The preset data length is determined according to the file acquisition request.

The file acquisition request comprises target file data with a preset length required by the upper layer application, and the target file data is data in the second target file. It can be seen that the target file data obtained by the upper layer application may be data of the whole or part of the second target file.

The target compression header includes a target compressed data size, i.e., a size of the data block, when the second target file is compressed. The target compression head also comprises a bitmap table, and the storage position of the target file data can be determined according to the bitmap table.

Please refer to fig. 10, which illustrates a flowchart of determining target document data according to an embodiment of the present application. Acquiring target file data corresponding to a second target file with a preset data length according to a target compression head, wherein the target file data comprises:

step 1001, according to the bitmap table in the target compression header, determines the target storage location of each data block in the target compressed data, and determines the target compressed data amount from the target compression header.

And the target compressed data is the compressed data of the second target file. The target compressed data includes a plurality of data blocks. The bitmap table in the target compression head comprises bitmap information of each data block, and the target storage position of each data block in the target compressed data after compression processing can be determined based on the bitmap information.

Step 1002, reading compressed data with a preset data length from a target storage location according to a target compressed data amount to obtain target data blocks, wherein the number of the target data blocks is one or more.

Correspondingly, the terminal can read the compressed data with the preset length from the target storage position, and specifically, the terminal can read the data blocks according to the size of the target compressed data amount to obtain the compressed data, wherein the obtained compressed data is a single or a plurality of target data blocks. Therefore, in the embodiment of the application, when the target file data is acquired, one block of data or a plurality of blocks of data in the second target file can be directly acquired, and the acquisition mode is flexible.

And step 1003, obtaining target file data according to the target data block.

As described above, in the storage process, the data block may be compressed into a compressed block according to the requirements of the upper layer application and then stored, or a part of the data block may not be compressed.

Therefore, for the target file data, after the target data block is obtained, it is necessary to determine whether the target data block is compressed and further obtain the target file data.

In one embodiment, obtaining target file data from the target data block includes: and determining whether the target data block needs to be decompressed according to the byte parameters of the target data block, and if the target data block needs to be decompressed, performing decompression on the target data block to obtain target file data.

Optionally, for the target data block, it may be determined whether the target data block needs to be a compressed block after compression processing according to the first 8 bytes of the target data block, and if yes, decompression processing needs to be performed to obtain the target file data. Among them, a decompression tool may be deployed in the ROMFS file system, and the decompression tool may be generated by a software algorithm. The terminal can decompress the target data block through the decompression tool to obtain target file data.

Optionally, the terminal further needs to configure a cache space in the process of reading the target file data, and after the target file data is obtained through decompression, the target file data can be stored in the cache space. And after the upper layer application reads the target file data from the cache space through the reading interface, the terminal can release the cache space.

In the embodiment of the application, after the ROMFS file system is started, the file in the ROMFS file system is read, the target file data is read according to the file acquisition request of the upper application, the data block is read from the FLASH memory space by block unit (for example, 4K or 64K) according to the compressed target compressed data corresponding to the second target file, decompression is performed, and the decompressed target file data is transmitted to the upper application.

For a clearer understanding of the storage process for compressing and decompressing files based on the ROMFS file system provided in the present application, the following is a general flow chart for the compression and decompression process.

Referring to fig. 11, a logic flow diagram of a file storage process according to an embodiment of the present application is shown. The ROMFS file system starts an image generation tool in the file storage process, reads files or folders under the folders to be stored, and records file header information of the folders into a linked list for the folders; for the file, judging whether the file needs to be compressed according to the passed parameters, namely the file header of the file, if so, compressing the file data in the file and storing the compressed data in a tmp temporary folder; of course, if it is determined that the file does not need to be compressed according to the file header of the file, the file header information of the file is sequentially recorded into the linked list, and the next file in the folder is read.

When all the files or folders in the folder are read, the mirror image file can be created. Wherein, the process of creating the mirror image file generates mirror image information such as a mirror image header mark of the mirror image file.

And reading the information of the files or the folders from the linked list in sequence according to the linked list. In the reading process, if a file is read, whether the file is a file needing to be compressed is judged based on a file header of the file, if so, the file header is written into the image file, the corresponding compressed block is read from the temporary folder according to the block, and the compressed block is written into the image file. In this process, bitmap information and the like of each compressed block are recorded.

When the reading of the file header information in the linked list is completed, the making of the mirror image file is completed, and the mirror image file can be further stored into a memory.

Please refer to fig. 12, which illustrates a logic flow diagram of a file reading process according to an embodiment of the present application.

After the ROMFS file system is powered on and started, the terminal opens the file in the file system to be acquired by the upper application through the open interface. And then reading the file header information of the file, and determining whether the file is subjected to compression processing or not based on the file header information.

If the file is determined to be compressed, reading the compression header information of the file, wherein the compression header information comprises information such as the compression type of the file, the original length size of the file, the block size during file compression and the like. And meanwhile, a BUF cache space is configured, wherein the BUF cache space comprises a BUF space for reading compressed data and a BUF space for storing decompressed data after the data are decompressed.

Further, the upper application in the terminal reads the specified data length from the BUF space of the compressed data through a read interface to obtain a plurality of data blocks. And for each data block, judging whether the data block needs to be decompressed according to the first 8 bytes of the data block. And if so, decompressing and storing the decompressed data in the BUF space of the decompressed data. And the terminal reads data through a read interface and sends the data to the upper application. And after the data with the required length is read, releasing the BUF space of the decompressed data, and finishing the file reading process.

In the embodiment of the application, for electronic equipment such as a camera, along with the development of the technology, the requirement on the pixel resolution of the camera is higher and higher, the parameters processed by an image processor chip in the electronic equipment are more and more, and the parameter files processed by the image processor chip are larger and larger. In the embodiment of the present application, compression processing may be performed on other files such as audio related files in the parameter file or other electronic devices. After the romfs file system is started, decompression is directly carried out when the file is read. Therefore, the storage cost can be reduced under the condition of storing more data files, and the storage space is saved.

It should be understood that, although the steps in the flowcharts related to the embodiments described above are shown in sequence as indicated by the arrows, the steps are not necessarily performed in sequence as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least a part of the steps in the flowcharts related to the embodiments described above may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the execution order of the steps or stages is not necessarily sequential, but may be performed alternately or alternately with other steps or at least a part of the steps or stages in other steps.

Based on the same inventive concept, the embodiment of the present application further provides a data storage device for implementing the above mentioned data storage method. The implementation scheme for solving the problem provided by the apparatus is similar to the implementation scheme described in the above method, so specific limitations in one or more embodiments of the data storage apparatus provided below may refer to the limitations on the data storage method in the foregoing, and details are not described here.

In one embodiment, as shown in fig. 13, there is provided a data storage device 1300 comprising: a first determination module 1301, a compression module 1302, and a storage module 1303, wherein:

a first determining module 1301, configured to determine, for a first object file to be stored in the ROMFS file system, whether compression processing needs to be performed on the first object file according to the type of the first object file and a file header of the first object file; the first target file comprises the file header and file data;

a compression module 1302, configured to determine a compression policy according to a storage space remaining in a memory when it is determined that compression processing needs to be performed on the first target file, and perform compression processing on the file data based on the compression policy to obtain compressed data corresponding to the first target file;

the storage module 1303 is configured to write the file header and the compressed data into an object image file, and store the object image file in the memory.

In an embodiment, the file header includes a file type parameter characterized by a plurality of bits, and the first determining module 1301 is specifically configured to: if the type of the first target file is a target type and the target bit in the file type parameter is a preset bit value, determining that the compression processing needs to be executed on the first target file; wherein the target type is a type that file data in the file is not empty.

In an embodiment, the compression module 1302 is specifically configured to: compressing the file data based on the compression strategy to obtain a compressed file, and generating a compression header, wherein the compression header comprises compression information related to compression; and taking the compressed file and the compression header as the compressed data.

In an embodiment, the compression module 1302 is specifically configured to: extracting a plurality of data blocks from the file data according to the compressed data amount; and respectively compressing each data block to obtain the compressed file.

In one embodiment, the header includes a file value, the file value being used to characterize the size of the first target file; the compression module 1302 is specifically configured to: obtaining a bitmap table; the bitmap table comprises bitmap information corresponding to each data block, and the bitmap information is used for representing the storage position of the data block; wherein, the bitmap information is generated in the process of writing the compressed data into the target image file; acquiring the file value according to the file header; and taking the bitmap table, the compressed data amount and the file value as the compression information.

In one embodiment, before writing the header and the compressed data into the target image file, the apparatus further comprises:

correspondingly, the storage module 1303 is specifically configured to: and writing the file header containing the compression value and the compression data into the target image file.

In an embodiment, the compressed data is stored in a temporary folder, and the storage module 1303 is specifically configured to: creating a target mirror image file; and reading the data blocks from the temporary folder according to the compressed data amount, and writing the read data blocks into the target image file in sequence.

In one embodiment, the apparatus further comprises:

an obtaining module, configured to determine whether the second target file performs compression processing according to the target file header; if the second target file is determined to execute the compression processing, reading a target compression head corresponding to the second target file; and acquiring target file data corresponding to the second target file with a preset data length according to the target compression head, wherein the preset data length is determined according to the file acquisition request.

In an embodiment, the obtaining module is specifically configured to: determining a target storage position of each data block in the target compressed data according to a bitmap table in the target compression head, and determining a target compressed data volume from the target compression head; reading compressed data with preset data length from the target storage position according to the target compressed data amount to obtain one or more target data blocks; and obtaining the target file data according to the target data block.

The various modules in the data storage device described above may be implemented in whole or in part by software, hardware, and combinations thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.

In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as shown in fig. 14. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The communication interface of the computer device is used for carrying out wired or wireless communication with an external terminal, and the wireless communication can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. The computer program is executed by a processor to implement a data storage method. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on the shell of the computer equipment, an external keyboard, a touch pad or a mouse and the like.

It will be appreciated by those skilled in the art that the configuration shown in fig. 14 is a block diagram of only a portion of the configuration associated with the present application, and is not intended to limit the computing device to which the present application may be applied, and that a particular computing device may include more or fewer components than shown, or may combine certain components, or have a different arrangement of components.

In one embodiment, an electronic device is further provided, which includes a memory and a processor, where the memory stores a computer program, and the processor implements the steps of the above method embodiments when executing the computer program.

In an embodiment, a computer-readable storage medium is provided, on which a computer program is stored which, when being executed by a processor, carries out the steps of the above-mentioned method embodiments.

In an embodiment, a computer program product is provided, comprising a computer program which, when being executed by a processor, carries out the steps of the above-mentioned method embodiments.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, database, or other medium used in the embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high-density embedded nonvolatile Memory, resistive Random Access Memory (ReRAM), magnetic Random Access Memory (MRAM), ferroelectric Random Access Memory (FRAM), phase Change Memory (PCM), graphene Memory, and the like. Volatile Memory can include Random Access Memory (RAM), external cache Memory, and the like. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), among others. The databases referred to in various embodiments provided herein may include at least one of relational and non-relational databases. The non-relational database may include, but is not limited to, a block chain based distributed database, and the like. The processors referred to in the various embodiments provided herein may be, without limitation, general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, or the like.

The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.

The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present application. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, and these are all within the scope of protection of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A method of data storage, the method comprising:

under the condition that the first target file needs to be compressed, determining a compression strategy according to the residual storage space in a memory, and compressing the file data based on the compression strategy to obtain compressed data corresponding to the first target file;

2. The method of claim 1, wherein the header includes a file type parameter characterized by a plurality of bits, and wherein determining whether a compression process needs to be performed on the first target file according to the type of the first target file and the header of the first target file comprises:

if the type of the first target file is a target type and a target bit in the file type parameter is a preset bit value, determining that the compression processing needs to be executed on the first target file; and the target type is a type that file data in the file is not empty.

3. The method of claim 2, wherein the target bit is a 0 th bit to a 2 nd bit in the file type parameter, and the preset bit values include a 0 th bit value of 0, a 1 st bit value of 1, and a 2 nd bit value of 1.

4. The method according to claim 1, wherein the compressing the file data based on the compression policy to obtain compressed data corresponding to the first target file comprises:

compressing the file data based on the compression strategy to obtain a compressed file, and generating a compression head, wherein the compression head comprises compression information related to compression;

and taking the compressed file and the compression header as the compressed data.

5. The method according to claim 4, wherein the compression policy includes a compression data amount, and compressing the file data based on the compression policy to obtain a compressed file comprises:

extracting a plurality of data blocks from the file data according to the compressed data amount;

and respectively compressing each data block to obtain the compressed file.

6. The method of claim 5, wherein the header includes a file value, the file value being used to characterize a size of the first target file; the generating a compression head includes:

obtaining a bitmap table; the bitmap table comprises bitmap information corresponding to each data block, and the bitmap information is used for representing the storage position of the data block; the bitmap information is generated in the process of writing the compressed data into the target image file;

acquiring the file value according to the file header;

the bit map table, the compressed data amount, and the file value are taken as the compression information.

7. The method of claim 6, wherein prior to writing the header and the compressed data into a target image file, the method further comprises:

determining a compression value of the compressed data, the compression value characterizing a size of the compressed data;

taking the compressed value as the file value;

correspondingly, the writing the file header and the compressed data into a target image file includes:

and writing the file header containing the compressed value and the compressed data into the target image file.

8. The method of claim 5, wherein the storing the compressed data in a temporary folder, and wherein the writing the header and the compressed data in a destination image file comprises:

creating a target mirror image file;

and reading data blocks from the temporary folder according to the compressed data amount, and writing the read data blocks into the target image file in sequence.

9. The method of claim 6, further comprising:

receiving a file acquisition request sent by an upper application, and determining a target file header corresponding to a second target file according to the file acquisition request;

determining whether the second target file executes compression processing according to the target file header;

if the second target file is determined to be subjected to compression processing, reading a target compression head corresponding to the second target file;

and acquiring target file data corresponding to the second target file with a preset data length according to the target compression head, wherein the preset data length is determined according to the file acquisition request.

10. The method according to claim 9, wherein the obtaining target file data corresponding to the second target file with a preset data length according to the target compression header comprises:

determining a target storage position of each data block in target compressed data according to a bitmap table in the target compression head, and determining a target compressed data volume from the target compression head;

reading compressed data with preset data length from the target storage position according to the target compressed data amount to obtain target data blocks, wherein the number of the target data blocks is one or more;

and obtaining the target file data according to the target data block.

11. The method of claim 10, wherein obtaining the target file data from the target data block comprises:

determining whether the target data block needs to be decompressed according to the byte parameters of the target data block;

and if the target data block is determined to need to be decompressed, decompressing the target data block to obtain the target file data.

12. A data storage device, the device comprising:

the compression module is used for determining a compression strategy according to the residual storage space in the storage under the condition that the first target file needs to be compressed, and compressing the file data based on the compression strategy to obtain compressed data corresponding to the first target file;

13. An electronic device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor realizes the steps of the method of any of claims 1 to 11 when executing the computer program.

14. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 11.

15. A computer program product comprising a computer program, characterized in that the computer program realizes the steps of the method of any one of claims 1 to 11 when executed by a processor.