CN116303114A

CN116303114A - Array compression processing method, device, equipment and storage medium

Info

Publication number: CN116303114A
Application number: CN202310257639.7A
Authority: CN
Inventors: 甘建旋
Original assignee: Apollo Intelligent Connectivity Beijing Technology Co Ltd
Current assignee: Apollo Intelligent Connectivity Beijing Technology Co Ltd
Priority date: 2023-03-09
Filing date: 2023-03-09
Publication date: 2023-06-23

Abstract

The disclosure provides an array compression processing method, device, equipment and storage medium, relates to the technical field of computers, and particularly relates to the technical field of cloud computing and big data. The specific implementation scheme is as follows: acquiring a plurality of memory address data contained in an array to be compressed; dividing an array to be compressed into an offset and positioning data corresponding to the memory address data according to the memory address data, wherein the offset is used for representing the same part among the memory address data, and the positioning data is used for representing different parts among the memory address data; obtaining a compressed array based on the positioning data corresponding to each of the memory address data; and obtaining a compressed array list based on the offset and the compressed array. The compression storage of the memory address data can be realized, the storage space occupied by the memory address array is greatly reduced, and the efficiency of index and memory addressing is not influenced.

Description

Array compression processing method, device, equipment and storage medium

Technical Field

The disclosure relates to computer technology, in particular to the technical field of cloud computing and big data, and especially relates to an array compression processing method, device, equipment and storage medium.

Background

With the development of cloud computing and big data technology, the memory used by a single server is larger (for example, a 64-bit operating system, and a big memory supporting 2++63=16777216 TB at maximum), up to hundreds of G or even T memories, and the occupied storage space is excessive.

However, the conventional Core Dump technology Core Dump is adopted to generate the memory image, and then the memory image is downloaded to the local area to meet the requirement of time cost, for example, the transmission time of the 200G memory image is up to 1 hour to several hours, and the time of analyzing the memory is not included, so that the analysis efficiency and the problem positioning solution are greatly reduced.

Disclosure of Invention

The present disclosure provides an array compression processing method, device, equipment and storage medium.

According to a first aspect of the present disclosure, there is provided an array compression processing method, the method including:

acquiring a plurality of memory address data contained in an array to be compressed;

dividing the array to be compressed into an offset and positioning data corresponding to the memory address data according to the memory address data, wherein the offset is used for representing the same part among the memory address data, and the positioning data is used for representing different parts among the memory address data;

Obtaining a compressed array based on the positioning data corresponding to each of the memory address data;

and obtaining a compressed array list based on the offset and the compressed array.

Further, the dividing the array to be compressed into the offset and the positioning data corresponding to the memory address data according to the memory address data includes:

determining the offset among the memory address data according to the memory address data;

and comparing the offset with the memory address data to obtain positioning data corresponding to the memory address data.

Further, the determining, according to the plurality of memory address data, an offset between the plurality of memory address data includes:

acquiring the same byte among a plurality of memory address data;

and obtaining the offset according to the same bytes among the memory address data.

Further, comparing the offset with the plurality of memory address data to obtain positioning data corresponding to the plurality of memory address data, respectively, including:

comparing the offset with a plurality of memory address data respectively to obtain different bytes among the memory address data;

And obtaining positioning data corresponding to each of the memory address data according to different bytes among the memory address data.

Further, before acquiring the plurality of memory address data included in the array to be compressed, the method further includes:

acquiring an array to be data sliced, wherein memory address data in the array to be data sliced exceeds the array capacity limit of the array to be compressed;

performing slicing processing on the array to be subjected to data slicing to obtain a sliced array meeting the array capacity limit of the array to be compressed;

and storing the array after the slicing processing into the array to be compressed, so that after the array to be compressed is full, a plurality of memory address data contained in the array to be compressed are obtained.

Further, after obtaining a compressed array list based on the offset and the compressed array, the method further includes:

and clearing the array to be compressed so that the cleared array to be compressed continuously receives new memory address data.

Further, the index value of the compressed array is consistent with the index value of the array to be compressed, and after obtaining the compressed array list based on the offset and the compressed array, the method further includes:

Responding to a memory address inquiry request, and acquiring an index value carried in the memory address inquiry request;

inquiring the compressed array list according to the index value to determine a target compressed array;

inquiring the target compressed array according to the index value to determine positioning data corresponding to the index value in the target compressed array;

and obtaining an array before compression based on the positioning data and the offset.

According to a second aspect of the present disclosure, there is provided an array compression processing apparatus, the apparatus including:

the data acquisition unit is used for acquiring a plurality of memory address data contained in the array to be compressed;

the dividing unit is used for dividing the array to be compressed into offset and positioning data corresponding to the memory address data according to the memory address data, wherein the offset is used for representing the same part among the memory address data, and the positioning data is used for representing different parts among the memory address data;

the first processing unit is used for obtaining a compressed array based on the positioning data corresponding to each of the memory address data;

And the second processing unit is used for obtaining a compressed array list based on the offset and the compressed array.

Further, the dividing unit includes:

the determining module is used for determining the offset among the memory address data according to the memory address data;

and the comparison module is used for respectively comparing the offset with the memory address data so as to obtain positioning data corresponding to the memory address data.

Further, the determining module is further configured to:

acquiring the same byte among a plurality of memory address data; and obtaining the offset according to the same bytes among the memory address data.

Further, the comparison module is further configured to:

comparing the offset with a plurality of memory address data respectively to obtain different bytes among the memory address data; and obtaining positioning data corresponding to each of the memory address data according to different bytes among the memory address data.

Further, the apparatus further comprises:

the array acquisition unit is used for acquiring an array of the to-be-data fragments, wherein the memory address data in the array of the to-be-data fragments exceeds the array capacity limit of the array to-be-compressed;

The slicing processing unit is used for carrying out slicing processing on the array of the data to be compressed so as to obtain a sliced array which meets the array capacity limit of the array to be compressed;

and the storage unit is used for storing the array after the slicing processing into the array to be compressed, so that after the array to be compressed is full, a plurality of memory address data contained in the array to be compressed are obtained.

Further, the index value of the compressed array is consistent with the index value of the array to be compressed, and the device further comprises:

the response unit is used for responding to the memory address query request and acquiring an index value carried in the memory address query request;

the first determining unit is used for inquiring the compressed array list according to the index value so as to determine a target compressed array;

the second determining unit is used for inquiring the target compressed array according to the index value so as to determine positioning data corresponding to the index value in the target compressed array;

and the third processing unit is used for obtaining an array before compression based on the positioning data and the offset.

According to a third aspect of the present disclosure, there is provided an electronic device comprising:

At least one processor; and

a memory communicatively coupled to the at least one processor; wherein,,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform any one of the methods.

According to a fourth aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the method according to any one of the claims.

According to a fifth aspect of the present disclosure, there is provided a computer program product comprising: a computer program stored in a readable storage medium, from which it can be read by at least one processor of an electronic device, the at least one processor executing the computer program causing the electronic device to perform the method of the first aspect.

According to the technology of the present disclosure, a plurality of memory address data contained in an array to be compressed are obtained; dividing an array to be compressed into an offset and positioning data corresponding to the memory address data according to the memory address data, wherein the offset is used for representing the same part among the memory address data, and the positioning data is used for representing different parts among the memory address data; obtaining a compressed array based on the positioning data corresponding to each of the memory address data; and obtaining a compressed array list based on the offset and the compressed array.

The present disclosure uses the same portion (same data, such as the same byte) between a plurality of memory address data as an offset, uses different portions (different data, such as different bytes) between a plurality of memory address data as positioning data when indexing, stores the positioning data corresponding to each of the plurality of memory address data to obtain a compressed array, and stores the offset and the compressed array into a compressed array list. Therefore, the embodiment of the disclosure can realize the compression storage of the memory address data, greatly reduce the storage space occupied by the memory address array, and simultaneously not affect the efficiency of index and memory addressing, thereby improving the memory analysis efficiency.

It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the disclosure, nor is it intended to be used to limit the scope of the disclosure. Other features of the present disclosure will become apparent from the following specification.

Drawings

The drawings are for a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:

FIG. 1 is a flow chart of an array compression processing method provided in accordance with an embodiment of the present disclosure;

FIG. 2 is a flow chart of an alternative array compression processing method provided in accordance with an embodiment of the present disclosure;

FIG. 3 is a flow chart of an alternative array compression processing method provided in accordance with an embodiment of the present disclosure;

FIG. 4 is a schematic diagram of a scenario in which an alternative array compression processing method of an embodiment of the present disclosure may be implemented;

FIG. 5 is a flow chart of an alternative array compression processing method provided in accordance with an embodiment of the present disclosure;

FIG. 6 is a schematic diagram of an array compression processing apparatus according to an embodiment of the present disclosure;

FIG. 7 is a schematic diagram of a framework of an alternative array compression processing apparatus provided in accordance with an embodiment of the present disclosure;

fig. 8 is a block diagram of an electronic device for implementing an array compression processing method according to an embodiment of the present disclosure.

Detailed Description

Exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding, and should be considered as merely exemplary. Accordingly, one of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

The terms referred to in this application are explained first:

a single server is a server providing all functions, and all clients are connected to the same server.

The Core Dump (Core Dump), sometimes referred to as a kernel in chinese, is a disk file that the operating system writes the contents of the process address space and other information about the state of the process when it receives certain signals to terminate operation. This information is often used for debugging when a process is to be aborted, and it is optional to save all the user space memory data of the process to disk, the file name usually being Core, which is called Core Dump.

Memory mapping: memory data used by an application program in an operating system during running is stored in a backup of a local file.

Java Development Kit, JDK for short, is a Java language software development kit, and is mainly used for Java application programs on mobile devices and embedded devices.

jhat is a virtual machine heap dump snapshot analysis tool that Java virtual machines are self-contained.

Hadoop Distributed File System, abbreviated as HDFS, is a distributed file system, and generally includes metadata nodes, namely a nalcode and a data node, wherein the nalcode mainly stores file metadata information, and the entire HDFS cannot be used when the nalcode fails.

At present, a virtual machine heap dump snapshot analysis tool jhat in a software development tool kit JDK adopting Java language can analyze and analyze the memory use condition of Java, feed back the thread stack of threads and the like. However, there is a problem in that since jhat directly loads a memory map into a memory and part of compressed data is decompressed, jhat cannot run for a program with a large memory, and thus, the memory image analysis capability for a memory image larger than 4G is limited or cannot be analyzed by jhat.

In addition, there is a way to analyze the memory image by using the program debug tool GDB, but since the GDB needs to partially load the whole process of executing the memory image restore program, the detailed analysis of the memory cannot be performed by using the GDB.

In order to solve the above problems, the present disclosure provides an array compression processing method, device, equipment and storage medium, which are applied to the technical fields of cloud computing and big data, wherein the array compression processing method is used for performing online analysis on a target server, and in general, a memory can be compressed to a compression rate of more than 60%, and meanwhile, the efficiency of index and memory addressing is not affected.

Example 1

Fig. 1 is a flowchart of an array compression processing method according to an embodiment of the present disclosure, as shown in fig. 1, the array compression processing method provided by the present disclosure includes the following method steps:

s101, acquiring a plurality of memory address data contained in an array to be compressed.

S102, dividing the array to be compressed into offset and positioning data corresponding to the memory address data according to the memory address data.

The offset is used for representing the same part among a plurality of the memory address data, and the positioning data is used for representing different parts among a plurality of the memory address data.

S103, obtaining a compressed array based on the positioning data corresponding to each of the memory address data.

S104, obtaining a compressed array list based on the offset and the compressed array.

In this disclosure, the array to be compressed may be an array including a plurality of memory address data, for example, an array already full of memory address data, and because the array full of memory address data occupies too much memory space, the array compression method of the embodiment of the disclosure may be used to perform array compression processing on the array to reduce the memory space occupied by the memory address array.

Alternatively, the plurality of memory address data may be all or part of the array to be compressed, and the memory address data may be the object memory address.

In one example, the memory of a physical machine server (i.e., a standalone server) is typically between 4G and several T. For some applications, most objects (type variables, examples after memory space is allocated, and a certain amount of memory is occupied) are usually small in size, several bytes or tens of bytes, and generally below several K, and 0x 0000-0 xffff of storage space can store objects with a size of 63K.

In one example, for a 64-bit operating system, the storage space that one memory address occupies is approximately 64 bits (8 bytes), e.g., two bytes of space in 0 xffffffffffffffffffff0000-0 xffffffffffffffffffffffffffffffx may hold objects of 63K size.

As shown in fig. 2, 0xffffffffffff 000111-0xffffffffff000631 is a to-be-compressed array formed by a plurality of memory address data, and each memory address data in the to-be-compressed array occupies 8 bytes, but there are 6 bytes whose sizes are all the same (offset: 0xffffffffff000 as shown in fig. 2), so that the target memory address can be directly compressed when the target memory address is saved: the same part (same data part, such as same byte) among the memory address data is used as offset, other different parts (such as different bytes) among the memory address data are used as positioning data, and each positioning data occupies only two bytes in the compressed array, so that the size occupied by each memory address data in the compressed array can be reduced by 6 bytes.

And then, respectively storing the positioning data corresponding to each of the memory address data into the compressed array to jointly form a compressed array list based on the offset and the compressed arrays (for example, the compressed array 1, the compressed array 2, the compressed array 3 and the like).

In the embodiment of the disclosure, the same portion (same data, such as the same byte) among the plurality of memory address data is used as an offset, and different portions (different data, such as different bytes) among the plurality of memory address data are used as positioning data when indexing, and then the positioning data corresponding to each of the plurality of memory address data is stored to obtain a compressed array, and the offset and the compressed array are stored in a compressed array list. Therefore, the embodiment of the disclosure can realize the compression storage of the memory address data, greatly reduce the storage space occupied by the memory address array, and simultaneously not affect the efficiency of index and memory addressing, thereby improving the memory analysis efficiency.

In another example, the compression processing method of the memory address array provided by the embodiment of the disclosure may be used for performing online analysis on the target server, where the compression rate of the memory may be up to 60% or more in general, and the efficiency of index and memory addressing is not affected. For example, if the memory address data stored in the original memory address array (i.e. the array to be compressed) occupies 16×8 bits of space, and the compressed array stored in the obtained compressed array only occupies 5×8 bits of space by adopting the embodiment of the present disclosure, therefore, the compression rate of the memory address array can reach 5/16=31.25 by adopting the array compression processing method provided by the embodiment of the present disclosure.

Example 2

According to one or more embodiments of the present disclosure, dividing an array to be compressed into an offset and positioning data corresponding to each of a plurality of memory address data according to the plurality of memory address data includes:

s201, determining the offset between the memory address data according to the memory address data.

S202, comparing the offset with the memory address data to obtain positioning data corresponding to the memory address data.

In the above embodiment, each memory address data includes: positioning data and offset corresponding to the memory address data.

As an alternative embodiment, for an array to be compressed that is full of 64-bit memory address data, the offset between multiple memory address data in the array to be compressed, that is, the same byte between multiple memory address data, may be calculated first, as shown in fig. 3, after the calculated offset (0 xffffffff 000), the offset is compared with all the memory address data in the array to be compressed to obtain different bytes between each memory address data and other memory address data, and then the different bytes between each memory address data and other memory address data are used as positioning data corresponding to each of the multiple memory address data, and finally, the multiple positioning data obtain the compressed array.

Because the positioning data is used for representing different bytes among a plurality of memory address data, the index value of the compressed array is consistent with the index value of the array to be compressed, and the embodiment of the disclosure adopts the compression processing mode, so that the quick index inquiry performance of the array can be reserved, and meanwhile, a certain compression ratio can be ensured.

According to one or more embodiments of the present disclosure, determining an offset between a plurality of memory address data from the plurality of memory address data includes:

s301, obtaining the same byte among a plurality of memory address data.

S302, obtaining the offset according to the same bytes among the memory address data.

As shown in fig. 2 or 3, the same byte between the plurality of memory address data is obtained, so that the same byte between the plurality of memory address data is used as the offset (0 xffffffffff 000) of the plurality of memory address data, or may be regarded as the offset between the plurality of positioning data in the compressed array.

In the same memory address array, the same byte portions among the plurality of memory address data are obviously not different byte portions among the plurality of memory address data, which is more important for memory analysis, indexing and memory addressing, but occupy too much memory space, so that the embodiment of the present disclosure can be adopted to extract the same bytes among the plurality of memory address data as offset, and each compressed array corresponds to an offset, i.e. the compressed array can be restored to an array to be compressed. And, extract the different byte among a plurality of memory address data, as the content needing to have array after compressing.

In the embodiment of the disclosure, the same byte in the array to be compressed is compressed into an offset, and different parts are used as emphasis to store, which can be used as index and memory addressing.

According to one or more embodiments of the present disclosure, comparing an offset with a plurality of memory address data to obtain positioning data corresponding to each of the plurality of memory address data, includes:

s401, comparing the offset with a plurality of memory address data respectively to obtain a plurality of different bytes among the memory address data.

S402, obtaining positioning data corresponding to each of the memory address data according to different bytes among the memory address data.

In an example, after extracting the same bytes between the plurality of memory address data as the offset, the offset and the plurality of memory address data may be compared respectively, so as to obtain different bytes between each memory address data except the same bytes and other memory address data, so as to use the different bytes between each memory address data except the same bytes and other memory address data as the positioning data corresponding to each memory address data.

Therefore, the embodiment of the disclosure can realize the compression storage of the memory address data, greatly reduce the storage space occupied by the memory address array, and simultaneously reserve the positioning data corresponding to each memory address data without influencing the efficiency of index and memory addressing.

Example 3

In one example, on applications such as big data and recommendation, since the memory of an application is large, 100G is less, T is more, and the memory information is only analyzed by analyzing on the target server, and is rarely downloaded. For example, as shown in fig. 4, in the HDFS application of the big data file system, a naminode application in a 200G memory stores metadata information of the file system. In order to analyze the memory structure and storage condition of the application, a memory mapping file may be generated by jmap, and the memory mapping file may be saved to the local area of the target server, for example, a memory mapping analysis module of the target server, so as to analyze the object structure of the application, for example, the version, signature, size, type, distribution, etc., and knowing these allocation information, the use condition of the obtained memory may be known. Conversely, the memory allocator may be optimized, e.g., how large block sizes are used in particular, how large the memory granularity per block is, etc.

In one example, when an application is running, it is preferable to apply for a memory creation object, where the memory allocator is required to apply for a memory of a corresponding size, and the memory allocator typically applies for a large continuous memory (e.g. 500M) from the system, allocates the memory block into small blocks (e.g. each small block 1K may be allocated 500 x 1024k=500M), and applies for a memory of a size of 100 bytes in one of the 1K memory blocks when an object (assuming 100 bytes) is created.

However, if one object is 4K, one small block is not enough to be used, and a plurality of small blocks are spliced together and reassigned to the applicant. If a 4K space is built up without several connected memories, the distributor cannot distribute the memories, and the program will go wrong. If the discontinuous available space exceeds 4K, errors occur, and if the situation is that the memory is broken, the memory cannot be used, and the utilization efficiency of the memory is affected. The size distribution of the object is known, that is, the memory required by the object is roughly known, so that the size of the memory block, the memory policy (memory reclamation) and the like can be configured in advance.

For a process of big data or recommendation and other big application, the occupied memory is quite large, the number of objects or variables reaches the scale of hundreds of millions or billions, if only one array is used for storing the array, it is not practical, because the array is continuous space, and only one array is used, the array is too large. Therefore, in this case, the array of large memory addresses can be changed into one array to be compressed by considering the array fragment, so that the continuous memory space is not occupied, and the offset during the compression process is larger, and the compression rate is higher.

According to one or more embodiments of the present disclosure, before acquiring the plurality of memory address data included in the array to be compressed, the method further includes:

s501, acquiring an array of to-be-data fragments, wherein memory address data in the array of to-be-data fragments exceeds the array capacity limit of the array to-be-compressed.

S502, slicing the array to be data sliced to obtain the sliced array meeting the array capacity limit of the array to be compressed.

S503, taking the array after the slicing processing as an array to be compressed, so that after the array to be compressed is full, a plurality of memory address data contained in the array to be compressed are obtained.

In an example, in the memory map parsing module of the target server, as shown in fig. 5, it may be determined whether the array stored with the memory address data belongs to the array to be fragmented, and if the memory address data of one memory address array exceeds the array capacity limit of the array to be compressed, the memory address array is the array to be fragmented. In the embodiment of the disclosure, the array to be fragmented may be fragmented by using an array fragment device, so as to obtain a fragmented array that meets the array capacity limitation of the array to be compressed.

In one embodiment, after the array after the fragmentation is used as the array to be compressed, it is determined whether the array to be compressed is full, so that after it is determined that the array to be compressed is full, a plurality of memory address data included in the array to be compressed is obtained, so as to execute the array compression processing method provided in the above embodiment of the present disclosure, for example, in steps S101 to S104.

In another embodiment, after determining that the array to be compressed is not full, the method continues to wait for inputting new memory address data until the array to be compressed is full, and then executes the array compression processing method provided in steps S101-S104.

By means of the method, in the embodiment of the disclosure, the large data fragment arrays can be processed into the one-by-one data fragment arrays to be compressed, so that continuous memory space is not occupied, and meanwhile, the offset in compression processing is larger, and the compression rate is higher.

In another example, after obtaining the compressed array list based on the offset and the compressed array, the method further includes: and clearing the array to be compressed so that the cleared array to be compressed continuously receives new memory address data.

It can be appreciated that, because the array to be compressed receives the externally stored memory address data, in order to realize continuous storage and large-capacity storage of the memory address data, in the embodiment of the present disclosure, array compression processing is performed on the already stored array to be compressed, then both the offset and the compressed array are stored into the compressed array list, in order to avoid repeated storage of the memory address data, after that, the array to be compressed can be emptied, so that the emptied array to be compressed continuously receives new memory address data, and further, continuous storage of more memory address data is realized, so that a large amount of occupation of the storage space can be avoided.

Example 4

Fig. 6 is a schematic flow chart of an alternative array compression processing method according to one or more embodiments of the present disclosure, where, as shown in fig. 6, an index value of a compressed array is consistent with an index value of an array to be compressed, and after obtaining a compressed array list based on an offset and the compressed array, the method further includes:

s601, responding to a memory address query request, and acquiring an index value carried in the memory address query request.

S602, inquiring the compressed array list according to the index value to determine the target compressed array.

S603, inquiring the target compressed array according to the index value to determine positioning data corresponding to the index value in the target compressed array.

S604, obtaining an array before compression based on the positioning data and the offset.

In the embodiment of the present disclosure, since the index value of the compressed array and the index value of the array to be compressed are identical, after the compressed array list is obtained, when the memory address in the compressed array is obtained, the position of the compressed array in the compressed array list may be indexed first, the corresponding target compressed array is found, then the corresponding positioning data is found in the corresponding target compressed array, and the pre-compressed array may be obtained based on the positioning data and the offset.

According to the embodiment of the disclosure, by adopting the compression processing mode, the quick index query performance of the array can be reserved, and meanwhile, a certain compression ratio can be ensured.

According to one or more embodiments, the array compression processing method provided by the embodiments of the present disclosure is adopted to obtain a plurality of memory address data contained in an array to be compressed; dividing an array to be compressed into offset and positioning data corresponding to the memory address data according to the memory address data; obtaining a compressed array based on the positioning data corresponding to each of the memory address data; and obtaining a compressed array list based on the offset and the compressed array.

The present disclosure addresses the data by taking the same byte between the plurality of memory address data as an offset and taking the different byte between the plurality of memory address data as the positioning data when indexing. And because of the positioning data corresponding to each of the memory address data, the compressed array is stored, and the index value of the compressed array is consistent with the index value of the array to be compressed. The compression processing mode not only reserves the quick index inquiry of the array, but also saves a certain compression ratio.

Therefore, the embodiment of the disclosure can realize the compression storage of the memory address data, greatly reduce the storage space occupied by the memory address array, and simultaneously not affect the efficiency of index and memory addressing, thereby improving the memory analysis efficiency.

In one example, an array has a plurality of elements (e.g., memory address data), each element having a sequence number for locating the element in the array, the sequence number being an index value, and the index value being used to find the corresponding element in the array. According to the embodiment of the disclosure, the index value of the compressed array is consistent with the index value of the array to be compressed, so that the compressed array can be quickly searched, the storage space can be saved by adopting an array compression processing mode, and the efficiency of data in the index search array is not influenced.

The scheme of the present disclosure can compress the original memory address array (i.e. the array to be compressed) in a certain proportion, thereby saving the occupied storage space, and the compression processing can not affect the quick index inquiry of the compressed array, thus having an important effect in the memory analysis direction.

In the technical scheme of the disclosure, the related processes of collecting, storing, using, processing, transmitting, providing, disclosing and the like of the personal information of the user accord with the regulations of related laws and regulations, and the public order colloquial is not violated.

Example 5

Fig. 7 is a schematic frame diagram of an array compression processing apparatus according to an embodiment of the present disclosure, and as shown in fig. 7, the present disclosure further provides an array compression processing apparatus 700, including:

the data acquisition unit 701 is configured to acquire a plurality of memory address data included in the array to be compressed.

The dividing unit 702 is configured to divide the array to be compressed into an offset and positioning data corresponding to each of the plurality of memory address data according to the plurality of memory address data, where the offset is used to represent the same portion between the plurality of memory address data, and the positioning data is used to represent different portions between the plurality of memory address data.

The first processing unit 703 is configured to obtain a compressed array based on the positioning data corresponding to each of the plurality of memory address data.

The second processing unit 704 is configured to obtain a compressed array list based on the offset and the compressed array.

The array compression processing device provided by the embodiment of the disclosure is adopted to acquire a plurality of memory address data contained in an array to be compressed; dividing an array to be compressed into offset and positioning data corresponding to the memory address data according to the memory address data; obtaining a compressed array based on the positioning data corresponding to each of the memory address data; and obtaining a compressed array list based on the offset and the compressed array.

According to one or more embodiments of the present disclosure, the above-described dividing unit includes:

and the determining module is used for determining the offset among the memory address data according to the memory address data.

According to one or more embodiments of the present disclosure, the above-described determination module is further configured to:

According to one or more embodiments of the present disclosure, the above-described alignment module is further configured to:

comparing the offset with the memory address data to obtain different bytes among the memory address data; and obtaining positioning data corresponding to the memory address data according to the different bytes among the memory address data.

According to one or more embodiments of the present disclosure, the above-described apparatus further includes:

the array acquisition unit is used for acquiring the array of the to-be-data fragments, wherein the memory address data in the array of the to-be-data fragments exceeds the array capacity limit of the array to-be-compressed.

And the slicing processing unit is used for carrying out slicing processing on the array to be data sliced so as to obtain a sliced array which meets the array capacity limit of the array to be compressed.

And the storage unit is used for storing the array after the slicing processing into the array to be compressed so as to acquire a plurality of memory address data contained in the array to be compressed after the array to be compressed is full.

According to one or more embodiments of the present disclosure, the index value of the compressed array is consistent with the index value of the array to be compressed, and the apparatus further includes:

and the response unit is used for responding to the memory address query request and acquiring the index value carried in the memory address query request.

And the first determining unit is used for inquiring the compressed array list according to the index value so as to determine a target compressed array.

And the second determining unit is used for inquiring the target compressed array according to the index value so as to determine positioning data corresponding to the index value in the target compressed array.

According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

According to an embodiment of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform a method according to any one of the above.

According to an embodiment of the present disclosure, there is provided a computer program product comprising: a computer program stored in a readable storage medium, from which at least one processor of an electronic device can read, the at least one processor executing the computer program causing the electronic device to perform the solution provided by any one of the embodiments described above.

The present disclosure also provides an electronic device, according to an embodiment of the present disclosure, fig. 8 shows a schematic block diagram of an example electronic device 800 that may be used to implement an embodiment of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the disclosure described and/or claimed herein.

As shown in fig. 8, the apparatus 800 includes a computing unit 801 that can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM) 802 or a computer program loaded from a storage unit 808 into a Random Access Memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other by a bus 804. An input/output (I/O) interface 805 is also connected to the bus 804.

Various components in device 800 are connected to I/O interface 805, including: an input unit 806 such as a keyboard, mouse, etc.; an output unit 807 such as various types of displays, speakers, and the like; a storage unit 808, such as a magnetic disk, optical disk, etc.; and a communication unit 809, such as a network card, modem, wireless communication transceiver, or the like. The communication unit 809 allows the device 800 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.

The computing unit 801 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of computing unit 801 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the respective methods and processes described above, such as an array compression processing method. For example, in some embodiments, the array compression processing method may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program may be loaded and/or installed onto device 800 via ROM 802 and/or communication unit 809. When a computer program is loaded into RAM 803 and executed by computing unit 801, one or more steps of the array compression processing method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the array compression processing method in any other suitable manner (e.g., by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), complex Programmable Logic Devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

Program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that the program code, when executed by the processor or controller, causes the functions/operations specified in the flowchart and/or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the internet.

The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical hosts and VPS service ("Virtual Private Server" or simply "VPS") are overcome. The server may also be a server of a distributed system or a server that incorporates a blockchain.

It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps recited in the present disclosure may be performed in parallel or sequentially or in a different order, provided that the desired results of the technical solutions of the present disclosure are achieved, and are not limited herein.

The above detailed description should not be taken as limiting the scope of the present disclosure. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present disclosure are intended to be included within the scope of the present disclosure.

Claims

1. An array compression processing method, the method comprising:

2. The method of claim 1, wherein the dividing the array to be compressed into the offset and the positioning data corresponding to the memory address data according to the memory address data comprises:

3. The method of claim 2, wherein the determining an offset between the plurality of memory address data from the plurality of memory address data comprises:

acquiring the same byte among a plurality of memory address data;

4. The method of claim 2, wherein comparing the offset with the plurality of memory address data to obtain positioning data corresponding to each of the plurality of memory address data, respectively, comprises:

5. The method of claim 1, wherein prior to retrieving the plurality of memory address data contained in the array to be compressed, the method further comprises:

6. The method of claim 1, wherein after deriving a list of compressed arrays based on the offset and the compressed arrays, the method further comprises:

7. The method of any of claims 1 to 6, wherein the index value of the compressed array and the index value of the array to be compressed are identical, the method further comprising, after deriving a list of compressed arrays based on the offset and the compressed array:

8. An array compression processing apparatus, the apparatus comprising:

9. The apparatus of claim 8, wherein the dividing unit comprises:

10. The apparatus of claim 9, wherein the means for determining is further for:

11. The apparatus of claim 9, wherein the alignment module is further to:

12. The apparatus of claim 8, wherein the apparatus further comprises:

13. The apparatus according to any one of claims 8 to 12, wherein the index value of the compressed array and the index value of the array to be compressed agree, the apparatus further comprising:

14. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

15. A non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-7.

16. A computer program product comprising a computer program which, when executed by a processor, implements the steps of the method of any of claims 1-7.