WO2023045204A1

WO2023045204A1 - Method and system for generating finite state entropy coding table, medium, and device

Info

Publication number: WO2023045204A1
Application number: PCT/CN2022/074614
Authority: WO
Inventors: 吴睿振; 秦臻
Original assignee: 苏州浪潮智能科技有限公司
Priority date: 2021-09-22
Filing date: 2022-01-28
Publication date: 2023-03-30
Also published as: CN113572479B; CN113572479A

Abstract

Provided are a method and a system for generating a finite state entropy coding table. The method comprises: obtaining frequencies of occurrence of characters in a data block to be coded; obtaining a quantity of states of a corresponding column on the basis of a preset maximum state value and the frequencies of occurrence, then forming an empty table of a finite state entropy coding table; on the basis of a row number of each space in the empty table and a frequency of occurrence of a corresponding character, obtaining an initial value of the space, thus obtaining an initial table; traversing the initial table to determine whether an initial value at an m-th row and an n-th column traversed is greater than an initial value at an (m-1)th row and the n-th column; if so, using the initial value at the m-th row and the n-th column as a temporary value thereof, and determining whether said initial value is repeated with a previous initial value; if not, using the temporary value thereof as a state value thereof; after traversing is completed, obtaining all state values, and determining whether a maximum state value exists; if so, generating a state table of the finite state entropy coding table on the basis of all the state values, thereby implementing hardware acceleration of finite state entropy coding.

Description

A method, system, medium and device for generating a finite state entropy coding table

This application claims the priority of the Chinese patent application submitted to the China Patent Office on September 22, 2021, with the application number 202111107199.4, and the title of the invention is "a method and system for generating a finite state entropy coding table", the entire content of which is by reference incorporated in this application.

technical field

The present application relates to the technical field of data compression, in particular to a method and system for generating a finite state entropy coding table.

Background technique

With the advent of the big data era, in specific application fields such as the Internet of Things and artificial intelligence, the low-latency requirements for massive data processing continue to increase, and lossless data compression technology is becoming more and more important. Lossless data compression can be divided into algorithm based on data statistics and algorithm based on dictionary according to the principle of compression. Among them, algorithms based on data statistics include Shannon-Fano coding, Huffman (Hufman) coding, arithmetic coding, run-length coding, and finite state entropy coding (finite state entropy, FSE), etc.; algorithms based on dictionaries include LZ77 (Lempel- Ziv77) encoding and LZ78 (Lempel-Ziv 78) encoding, etc.

In order to improve universality, data compression schemes usually use two or more compression algorithms for mixed compression. The fast lossless compression algorithm Zstd (Zstandard) is a hybrid compression algorithm composed of LZ77 encoding, Hufman encoding and FSE. Compared with other compression algorithms (such as Deflate algorithm, Bzip2 algorithm, Brotli algorithm), Zstd has better compression performance. In addition, Zstd is an open source algorithm that provides 22 compression levels for weighing compression speed and compression rate, and has been widely used in Linux kernel, FreeBSD operating system, and AWS Redshift data warehouse and other fields.

The lossless compression technology based on software implementation has the advantages of high flexibility, universality and low cost, but the software execution method can only be executed sequentially, resulting in the central processing unit (CPU) being occupied for a long time when processing massive data. , so that the compression speed is greatly reduced, and it is difficult to meet the demand for real-time compression processing of massive data in specific application fields. Using hardware to implement is an effective way to solve the above problems. Benefiting from the inherent parallel processing characteristics of hardware, it can achieve the purpose of improving transmission speed, resource utilization and security.

The statistical analysis of the three main components in Zstd: LZ77, FSE and Huffman shows that the ratio of compression time is about 4:1:1. Although the proportion of FSE is not large, it has a greater impact on the performance of Zstd. Due to the use of FSE, Zstd has better compression performance than other hybrid compression algorithms. In addition, hardware acceleration schemes for LZ77 and Huffman coding are relatively mature, but FSE, as a new type of compression algorithm, has both the accuracy similar to arithmetic coding and the compression speed of Huffman coding, and the recoding of symbols can be accurate to decimals bit, multiplication and division do not need to be used in the computation to update the state. Therefore, researching the hardware acceleration architecture of FSE is of great significance to realize the overall acceleration of the Zstd algorithm, and it is also an effective method to meet the needs of specific application fields.

Contents of the invention

In view of this, the purpose of this application is to propose a method and system for generating a finite state entropy coding table, so as to solve the problem in the prior art that there is no hardware acceleration solution for finite state entropy coding.

Based on the above purpose, the application provides a method for generating a finite state entropy coding table, comprising the following steps:

The frequency of occurrence of each character in the data block is obtained based on the proportion of each character in the data block to be encoded;

Based on the preset maximum state value and the frequency of occurrence of each character, the state quantity of the corresponding column of each character in the finite state entropy coding table is obtained, and the space of the finite state entropy coding table is formed based on the state quantity of the corresponding column of each character and the frequency of occurrence of each character surface;

The initial value of each blank is obtained based on the number of rows of each blank in the empty table and the frequency of occurrence of the corresponding character, and the initial table of the finite state entropy coding table is obtained based on the filling of the initial value of each blank;

Traverse the initial table, and judge whether the initial value of the nth column of the mth row traversed is greater than the initial value of the nth column of the m-1th row;

Responding to the fact that the initial value of the nth column of the mth row is greater than the initial value of the m-1th row's nth column, the initial value of the mth row's nth column is used as its temporary value, and the temporary value of the mth row's nth column is judged Is it repeated with the initial value arranged before it;

Responding to the fact that the temporary value of row m and column n does not overlap with the initial value arranged before it, the temporary value of row m and column n is used as its state value;

In response to the completion of traversal, obtain all state values of the finite state entropy coding table, and judge whether there is a maximum state value among all state values;

In response to having a maximum state value, a state table of finite state entropy encoding tables is generated based on all state values.

In some embodiments, the method also includes:

In response to the initial value of the nth column of the mth row being less than or equal to the initial value of the m-1th row of the nth column, the value after the initial value of the m-1th row of the nth column is increased by the preset increment value as the mth row Temporary value for column n.

In some embodiments, the method also includes:

In response to the repetition of the temporary value of row m and column n with the initial value arranged before it, increasing the temporary value of row m and column n by a preset increment value as the updated temporary value until the updated temporary value It is not repeated with the initial value arranged before it, and the updated temporary value is used as the state value of the mth row and the nth column.

In some embodiments, the method also includes:

In response to no maximum state value, the largest numerical value among all state values is replaced by the maximum state value to generate a state table of the finite state entropy encoding table.

In some embodiments, based on the preset maximum state value and the frequency of occurrence of each character, the number of states in the columns corresponding to each character in the finite state entropy coding table includes:

The preset maximum state value is multiplied by the frequency of occurrence of each character to respectively obtain the state quantity of the column corresponding to each character in the finite state entropy coding table.

In some embodiments, forming an empty table of the finite state entropy encoding table based on the number of states of each character's corresponding column and the frequency of occurrence of each character includes:

The number of states of each character is used as the number of spaces in the corresponding column of the finite state entropy coding table, and the order between the columns is arranged based on the frequency of occurrence of each character to form an empty table of the finite state entropy coding table .

In some embodiments, arranging the order of the columns based on the frequency of occurrence of each character includes:

Arrange the corresponding columns from left to right according to the descending order of the occurrence frequency of each character.

In some embodiments, obtaining the initial value of each blank based on the number of rows of each blank in the empty table and the frequency of occurrence of the corresponding character includes:

The ratio of the number of rows where each space is located in the empty table to the frequency of occurrence of the corresponding character is rounded towards zero to obtain the initial value of each space.

In some embodiments, traversing the initial table includes:

The initial table is traversed in order from left to right and top to bottom.

In some embodiments, m is an integer greater than or equal to 2, and n is an integer greater than or equal to 1.

In another aspect of the present application, a system for generating a finite state entropy coding table is also provided, including:

The frequency of occurrence obtaining module is configured to obtain the frequency of occurrence of each character in the data block based on the proportion of each character in the data block to be encoded;

Empty table obtaining module, configured to obtain the state quantity of each character corresponding column in the finite state entropy coding table based on the preset maximum state value and the frequency of occurrence of each character, and based on the state quantity of each character corresponding column and the frequency of occurrence of each character form an empty list of finite state entropy encoding tables;

The initial table obtaining module is configured to obtain the initial value of each blank based on the number of rows of each blank in the empty table and the frequency of occurrence of the corresponding character, and obtain the initial table of the finite state entropy coding table based on the filling of the initial value of each blank ;

The first judging module is configured to traverse the initial table, and judge whether the initial value of the mth row and the nth column traversed is greater than the initial value of the m-1th row and the nth column;

The second judging module is configured to respond to the initial value of the nth column of the mth row being greater than the initial value of the m-1th row of the nth column, taking the initial value of the mth row of the nth column as its temporary value, and judging the first Whether the temporary value of column n in row m is repeated with the initial value arranged before it;

A state value module configured to use the temporary value of the nth row of the mth row as its state value in response to the fact that the temporary value of the mth row and the nth column is not repeated with the initial value arranged before it;

The third judging module is configured to obtain all the state values of the finite state entropy coding table in response to the completion of the traversal, and judge whether there is a maximum state value among all the state values; and

A state table generating module configured to generate a state table of a finite state entropy encoding table based on all state values in response to having a maximum state value.

Taking the number of states of each character as the number of spaces in the corresponding column of the finite state entropy coding table, and arranging the order between the columns based on the frequency of occurrence of each character to form the finite state entropy coding An empty list of tables.

In yet another aspect of the present application, a computer-readable storage medium is provided, which stores computer program instructions, and implements any one of the above-mentioned methods when the computer program instructions are executed by a processor.

In yet another aspect of the present application, a computer device is provided, including a memory and a processor, where a computer program is stored in the memory, and when the computer program is executed by the processor, any one of the above-mentioned methods is executed.

The application at least has the following beneficial technical effects:

1. The method for generating the finite state entropy coding table of the present application has a simple and convenient data storage form, which can be stored in the form of a one-dimensional array without using a linked list similar to the Huffman tree binary tree, which can greatly reduce the memory space;

2. The generation method of the finite state entropy encoding table of the present application is applicable to the FSE compression and decompression implemented by hardware under the Zstd specification standard, mainly using a comparator and an adder, and the calculation is relatively simple, thereby effectively reducing hardware resource overhead and improving hardware utilization;

3. The speed of compression and decompression of the Zstd algorithm is further improved to meet the increasing demand for compression performance in specific application fields.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present application, and those skilled in the art can obtain other embodiments according to these drawings without creative efforts.

FIG. 1 is a schematic diagram of a method for generating a finite state entropy coding table provided according to an embodiment of the present application;

FIG. 2 is a schematic diagram of a data block to be encoded according to an embodiment of the present application;

Fig. 3 is the schematic diagram of the FSE Table initial table provided according to the embodiment of the present application;

Fig. 4 is a schematic diagram of the FSE Table calculation traversal process provided according to the embodiment of the present application;

Fig. 5 is the schematic diagram of the FSE Table iterative calculation I provided according to the embodiment of the application;

Fig. 6 is the schematic diagram of the FSE Table iterative calculation II provided according to the embodiment of the application;

Fig. 7 is the schematic diagram of the FSE Table that traversal is completed according to the embodiment of the present application;

Fig. 8 is the schematic diagram of the FSE Table state table provided according to the embodiment of the present application;

FIG. 9 is a schematic diagram of a system for generating a finite state entropy coding table according to an embodiment of the present application;

FIG. 10 is a schematic diagram of a computer-readable storage medium for implementing a method for generating a finite state entropy coding table according to an embodiment of the present application;

FIG. 11 is a schematic diagram of a hardware structure of a computer device implementing a method for generating a finite state entropy coding table according to an embodiment of the present application.

Detailed ways

In order to make the purpose, technical solution and advantages of the present application clearer, the embodiments of the present application will be further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

It should be noted that all expressions using "first" and "second" in the embodiments of the present application are to distinguish between two non-identical entities or non-identical parameters with the same name. It can be seen that "first", "second "two" is only for the convenience of expression, and should not be understood as a limitation to the embodiment of the present application. Furthermore, the terms "comprising" and "having", as well as any variations thereof, are intended to cover a non-exclusive inclusion, for example, of a process, method, system, product or other steps or elements inherent in a process, method, system, product, or device comprising a series of steps or elements.

Based on the above purpose, the first aspect of the embodiments of the present application proposes an embodiment of a method for generating a finite state entropy coding table. FIG. 1 is a schematic diagram of an embodiment of a method for generating a finite state entropy coding table provided by the present application. As shown in Figure 1, the embodiment of the present application includes the following steps:

Step S10, obtain the frequency of occurrence of each character in the data block based on the proportion of each character in the data block to be encoded;

Step S20, based on the preset maximum state value and the frequency of occurrence of each character, the number of states in the column corresponding to each character in the finite state entropy coding table is obtained, and a finite state entropy code is formed based on the number of states in the column corresponding to each character and the frequency of appearance of each character empty list of tables;

Step S30, obtain the initial value of each blank based on the number of rows of each blank in the empty table and the frequency of occurrence of the corresponding character, and obtain the initial table of the finite state entropy coding table based on filling the initial value of each blank;

Step S40, traversing the initial table, and judging whether the initial value of the mth row and the nth column traversed is greater than the initial value of the m-1th row and the nth column;

Step S50, in response to the fact that the initial value of the nth column of the mth row is greater than the initial value of the m-1th row of the nth column, the initial value of the mth row of the nth column is used as its temporary value, and the mth row of the nth column is judged Whether the temporary value of is a duplicate of the initial value arranged before it;

Step S60, in response to the fact that the temporary value of row m and column n does not overlap with the initial value arranged before it, take the temporary value of row m and column n as its state value;

Step S70, in response to the completion of the traversal, obtain all state values of the finite state entropy coding table, and determine whether there is a maximum state value among all state values;

Step S80, in response to having the largest state value, generate a state table of the finite state entropy coding table based on all state values.

In the embodiment of the present application, m is an integer greater than or equal to 2, and n is an integer greater than or equal to 1.

FSE belongs to an entropy encoding of tANS (table asymmetric numerical systems) in asymmetric numerical systems (asymmetric numerical systems, ANS). Existing research mainly focuses on tANS and uABS (uniform asymmetric binary systems), an important part of ANS, while there is little research on FSE hardware architecture.

The method for generating the finite state entropy coding table of the embodiment of the present application has a simple and convenient data storage form, and only uses a one-dimensional array to store, without using a Huffman tree binary tree linked list storage, which can greatly reduce the memory space; In addition, the generation method of the finite state entropy encoding table in the embodiment of the present application is suitable for hardware-implemented FSE (finite state entropy encoding) compression and decompression under the Zstd (fast lossless compression algorithm) standard, mainly using comparators and adders , the calculation is relatively simple, thereby effectively reducing hardware resource overhead and improving hardware utilization; further improving the compression and decompression speed of the Zstd algorithm, and meeting the increasing demand for compression performance in specific application fields.

All calculations and storage methods involved in the generation method of the finite state entropy encoding table in the embodiment of the present application can not only be implemented as hardware, but also can improve the efficiency of software calculations. Various implementation forms make the application more flexible. If it is realized by hardware, it can become a hardware acceleration technology for network data storage, which can accelerate the compression of data based on limited entropy encoding and effectively reduce the load on the server CPU (central processing unit). It can focus on data compression acceleration to help improve the performance of data centers.

In some embodiments, the method further includes: increasing the initial value of the m-1th row and the nth column by a preset The value after the increment value is used as the temporary value of the mth row and the nth column.

In an optional embodiment, the preset increment value is an integer greater than 0.

In some embodiments, the method further includes: in response to the temporary value of the mth row and the nth column being repeated with the initial value arranged before it, increasing the mth row mth column nth temporary value by a preset increment value as the updated until the updated temporary value does not overlap with the initial value arranged before it, and the updated temporary value is used as the state value of the mth row and the nth column.

In the above embodiment, the value of each space traversed will be compared with the value of the previous row in the same column and the query of the repeated value arranged before, and then the corresponding operation will be performed based on the comparison result to meet the corresponding conditions, and based on the query result Do the corresponding operation to meet the corresponding condition before traversing the next space.

In some embodiments, the method further includes: in response to no maximum state value, replacing the largest numerical value among all state values with the maximum state value to generate a state table of the finite state entropy encoding table.

In some embodiments, obtaining the number of states in the column corresponding to each character in the finite state entropy coding table based on the preset maximum state value and the frequency of occurrence of each character includes: multiplying the preset maximum state value and the frequency of occurrence of each character, To respectively obtain the state quantity of the column corresponding to each character in the finite state entropy coding table.

In some embodiments, forming an empty table of the finite state entropy coding table based on the number of states of the corresponding column of each character and the frequency of occurrence of each character includes: using the number of states of each character as the space of the corresponding column of the finite state entropy coding table number, and arrange the order among the columns based on the frequency of occurrence of each character to form an empty table of the finite state entropy coding table.

In some embodiments, arranging the order of the columns based on the frequency of occurrence of each character includes: arranging the corresponding columns from left to right in order of the frequency of occurrence of each character from large to small.

In some embodiments, obtaining the initial value of each blank based on the number of rows of each blank in the empty table and the frequency of occurrence of the corresponding character includes: calculating the number of rows of each blank in the empty table and the frequency of occurrence of the corresponding character The ratio of is rounded towards zero to get the initial value of each space.

In some embodiments, traversing the initial table includes: traversing the initial table in order from left to right and from top to bottom.

Specifically, the generation method of the finite state entropy coding table of an exemplary embodiment of the present application is as follows:

The first step: count and sort the frequency of occurrence of Symbol (character) in the data block Text to be encoded as shown in Figure 2, the following table can be obtained:

Table 1 Statistical table of character frequency of coded data to be compressed

字符character	aa	bb	cc	dd	ee	ff
出现次数The number of occurrences	55	99	1212	1313	1616	4545
出现频率Frequency of occurrence	0.050.05	0.090.09	0.120.12	0.130.13	0.160.16	0.450.45

The second step: The encoding and decoding process of FSE (Finite State Entropy, finite state entropy encoding) needs to use the core table FSE Table, the primary task of its generation method is to initially calculate the state of each symbol corresponding column in the finite state entropy encoding table Number:

The calculation method of the number of states is the product of the maximum number of states and the frequency of occurrence of each symbol. Here, the maximum number of states is taken as 31 (the larger the number of states in practical applications, the closer the compression effect will be to the limit compression rate).

That is, the number of states of Symbol x in the FSE Table is:

Num _{stat_x} ＝(Sum _stat ·P(S _x )) _round

Then the number of states in the Symbol a column is:

Num _{stat_a} ＝(Sum _stat P(S _a )) _round ＝(31*0.05) _round ＝(1.55) _round ＝1

The number of corresponding columns in the state table of other Symbol b, c, d, e and f columns is:

Num _{stat_b} = (Sum _stat P(S _b )) _round = (31*0.09) _round = 2

Num _{stat_c} ＝(Sum _stat P(S _c )) _round ＝(31*0.12) _round ＝3

Num _{stat_d} = (Sum _stat P(S _d )) _round = (31*0.13) _round = 4

Num _{stat_e} = (Sum _stat P(S _e )) _round = (31*0.16) _round = 4

Num _{stat_f} ＝(Sum _stat P(S _f )) _round ＝(31*0.45) _round ＝13

Step 3: According to the number of states in the column where each Symbol is located in the FSE Table calculated in the previous step, and the frequency of occurrence of each Symbol, an empty table with a determined size and shape can be formed. Then calculate the value in the space of each row separately, and the calculation rules for the value of each cell are as follows:

Val _{Num_row,x} = (Num_row/P(S _f )) _round

For the first row, the element at position (1, f) = 1/0.45 = 2.222..., rounded to zero is 2; the value at position (1, e) = 1/0.16 = 6.25, rounded to zero is 6, (1,d)=1/0.13=7.692..., rounded towards zero to be 7,.... The second line (2, f)=2/0.45=4.4444, which is approximately 4, ..., thus constructing the FSE Table initial table as shown in Figure 3.

Step 4: Adjust the values of the elements in the initial table. The entropy coded state table needs to satisfy the following two characteristics: (1) each value in the table is unique (that is, there is no duplication); (2) each column is sorted according to the value from small to large. Now adjust the initial table based on the traversal comparison method, and the traversal order is from left to right and from top to bottom, that is, the schematic diagram of the FSE Table calculation traversal process shown in Figure 4.

Assuming that the element a _m,n is searched, that is, the element whose position is (m,n) in the table, the judgment and processing of condition 1 are performed first, and then the judgment and processing of condition 2 are performed, and the judgment of condition 2 should traverse all a _m,n search the previous element namely

Condition 1 judgment and processing (assuming the default increment value is 1):

like

where m≥2

have a _m,n ≤a _m-1,n

Then a _m,n = a _m-1,n +1

Judgment and processing of condition 2 (assuming that the default increment value is 1):

like

in

have

but

This step is applied in this exemplary embodiment, the schematic diagram of FSE Table iterative calculation I as shown in Figure 5, when traversing to the element 6 of the third row of the f column, it is found that it is larger than the element 4 of the previous row, that is, the condition is carried out 2. Judgment and handling. It is found that element 6 in the third row of column f is repeated with element 6 in the first row of column e, and the element in the third row of column f is replaced by 7; after that, it is found that element 7 in the first row of column d is replaced by 7 , replace the element in the third row of column f with 8; after that, it is found that the element 8 in the first row of column c is repeated after replacing it with 8, and the element in the third row of column f is replaced with 9; it is found that after replacing it with 9, it is no longer If there is duplication, the elements in the third row of column f are replaced.

The schematic diagram of FSE Table iterative calculation II shown in Figure 6, when traversing to the fourth row of column f, its value is 8 less than the third row 9 of column f, since each column must be in an increasing state, the first The value of the three lines plus 1 is assigned to the fourth line, that is, the value of the fourth line is replaced with 9+1=10; after replacing with 10, the previous elements are traversed, and if there is no overlap, the replacement is complete.

Use this method to traverse all the elements of the operation table to get the schematic diagram of the traversed FSE Table as shown in Figure 7, where the bold font is the adjusted value.

Step 5: Arrange the maximum status value in the FSE Table. This operation is to insert the maximum status value 31 in this example into the FSE Table, and this maximum value must exist in the FSE Table. Observe that 31 does not appear in the currently generated FSE Table, and there is a maximum value of 30. At this time, replace 30 with 31, and the schematic diagram of the FSE Table status table shown in Figure 8 is obtained.

In a second aspect of the embodiments of the present application, a system for generating a finite state entropy coding table is also provided. FIG. 9 is a schematic diagram of an embodiment of a system for generating a finite state entropy coding table provided by the present application. As shown in Figure 9, a generation system of a finite state entropy encoding table includes: an occurrence frequency obtaining module 10 configured to obtain the proportion of each character in the data block based on the number ratio of each character in the data block to be encoded. Frequency of occurrence in the block; Empty table acquisition module 20, configured to obtain the state quantity of each character corresponding column in the finite state entropy coding table based on the preset maximum state value and the frequency of occurrence of each character, and based on the state of each character corresponding column Quantity and the frequency of occurrence of each character form the empty table of finite state entropy coding table; Initial table obtains module 30, is configured to obtain the initial value of each space based on the number of rows where each space is located in the empty table and the frequency of occurrence of the corresponding character , and obtain the initial table of the finite state entropy coding table based on the filling of the initial value of each space; the first judging module 40 is configured to traverse the initial table, and judge whether the initial value of the mth row nth column traversed is greater than the mth − The initial value of the nth column of row 1; the second judging module 50, configured to respond to the initial value of the nth column of the mth row being greater than the initial value of the m-1th row of the nth column, the mth row of the nth column The initial value is used as its temporary value, and it is judged whether the temporary value of the mth row and the nth column is repeated with the initial value arranged before it; the state value module 60 is configured to respond to the mth row and the nth column's temporary value and arrangement The initial value before it is not repeated, and the temporary value of the mth row and the nth column is used as its state value; the third judging module 70 is configured to obtain all state values of the finite state entropy coding table in response to the completion of the traversal, and judging whether there is a maximum state value among all state values; and a state table generation module 80 configured to generate a state table of a finite state entropy encoding table based on all state values in response to the presence of a maximum state value.

In the third aspect of the embodiment of the present application, a computer-readable storage medium is also provided. FIG. 10 shows a schematic diagram of a computer-readable storage medium for implementing a method for generating a finite state entropy encoding table according to an embodiment of the present application. . As shown in FIG. 10 , computer readable storage medium 3 stores computer program instructions 31 . When the computer program instructions 31 are executed by the processor, the method of any one of the above-mentioned embodiments is realized.

It should be understood that all the embodiments, features and advantages described above for the generation method of the finite-state entropy coding table according to the present application are equally applicable to the generation system of the finite-state entropy coding table according to the present application without conflicting with each other. and storage media.

The fourth aspect of the embodiment of the present application also provides a computer device, including a memory 402 and a processor 401 as shown in FIG. 11 , the memory 402 stores a computer program, and the computer program is executed by the processor 401 When implementing the method of any one of the above embodiments.

As shown in FIG. 11 , it is a schematic diagram of a hardware structure of an embodiment of a computer device implementing a method for generating a finite state entropy coding table provided by the present application. Taking the computer equipment shown in FIG. 11 as an example, the computer equipment includes a processor 401 and a memory 402 , and may also include: an input device 403 and an output device 404 . The processor 401, the memory 402, the input device 403, and the output device 404 may be connected through a bus or in other ways, and connection through a bus is taken as an example in FIG. 11 . The input device 403 can receive input numbers or character information, and generate key signal input related to user settings and function control of the generation system of the finite state entropy coding table. The output device 404 may include a display device such as a display screen.

The memory 402, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs and modules, such as the generation of the finite state entropy coding table in the embodiment of the present application The program instruction/module to which the method corresponds. The memory 402 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application program required by at least one function; the data storage area may store the data created by using the method for generating the finite state entropy coding table wait. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 402 may optionally include memory that is remotely located relative to the processor 401, and these remote memories may be connected to the local module through a network. Examples of the aforementioned networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The processor 401 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 402, that is, realizes the generation method of the finite state entropy encoding table in the above method embodiment.

Finally, it should be noted that the computer-readable storage medium (eg, memory) herein may be a volatile memory or a nonvolatile memory, or may include both volatile memory and nonvolatile memory. By way of example and not limitation, nonvolatile memory can include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory memory. Volatile memory can include random access memory (RAM), which can act as external cache memory. By way of example and not limitation, RAM is available in various forms such as Synchronous RAM (DRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). Storage devices of the disclosed aspects are intended to include, but are not limited to, these and other suitable types of memory.

Those of skill would also appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described generally in terms of their functionality. Whether such functionality is implemented as software or as hardware depends upon the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the functions in various ways for each specific application, but such implementation decisions should not be interpreted as causing a departure from the scope disclosed in the embodiments of the present application.

The above are the exemplary embodiments disclosed in the present application, but it should be noted that various changes and modifications can be made without departing from the scope of the embodiments disclosed in the present application defined by the claims. The functions, steps and/or actions of the method claims in accordance with the disclosed embodiments described herein need not be performed in any particular order. In addition, although the elements disclosed in the embodiments of the present application may be described or required in an individual form, they may also be understood as plural unless explicitly limited to a singular number.

It should be understood that as used herein, the singular form "a" and "an" are intended to include the plural forms as well, unless the context clearly supports an exception. It should also be understood that "and/or" as used herein is meant to include any and all possible combinations of one or more of the associated listed items. The serial numbers of the embodiments disclosed in the above-mentioned embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

Those of ordinary skill in the art should understand that: the discussion of any of the above embodiments is exemplary only, and is not intended to imply that the scope (including claims) disclosed by the embodiments of the present application is limited to these examples; under the idea of the embodiments of the present application , the technical features in the above embodiments or different embodiments can also be combined, and there are many other changes in different aspects of the above embodiments of the present application, which are not provided in details for the sake of brevity. Therefore, within the spirit and principle of the embodiments of the present application, any omissions, modifications, equivalent replacements, improvements, etc., shall be included in the protection scope of the embodiments of the present application.

Claims

A method for generating a finite state entropy coding table, comprising the following steps:

The frequency of occurrence of each character in the data block is obtained based on the proportion of each character in the data block to be encoded in the data block;

Based on the preset maximum state value and the frequency of occurrence of each character, the state quantity of the corresponding column of each character in the finite state entropy coding table is obtained, and the finite state entropy coding table is formed based on the state quantity of the corresponding column of each character and the frequency of occurrence of each character the empty list;

The initial value of each blank is obtained based on the number of rows where each blank is located in the empty table and the frequency of occurrence of the corresponding character, and the initial table of the finite state entropy coding table is obtained based on filling the initial value of each blank;

Traversing the initial table, judging whether the initial value of the nth column of the mth row traversed is greater than the initial value of the nth column of the m-1th row;

In response to the initial value of the n-th row of the m-th row being greater than the initial value of the m-1-th row of the n-th column, the initial value of the m-th row of the n-th column is used as its temporary value, and the m-th row is judged Whether the temporary value of the nth column is repeated with the initial value arranged before it;

In response to the fact that the temporary value of the mth row and the nth column does not overlap with the initial value arranged before it, the temporary value of the mth row and the nth column is used as its state value;

In response to the completion of the traversal, obtain all state values of the finite state entropy coding table, and determine whether there is the maximum state value among all the state values;

In response to having the largest state value, a state table of the finite state entropy encoding table is generated based on all state values.
The method according to claim 1, further comprising:

In response to the initial value of the nth row of the mth column being less than or equal to the initial value of the m-1th row of the nth column, the initial value of the m-1th row of the nth column is increased by a value after a preset increment value as a temporary value for the mth row and nth column.
The method according to claim 2, further comprising:

In response to the temporary value of the mth row and nth column being repeated with the initial value arranged before it, increasing the mth row and nth column’s temporary value by the preset incremental value as an updated temporary value, Until the updated temporary value does not overlap with the initial value arranged before it, the updated temporary value is used as the state value of the mth row and nth column.
The method according to claim 1, further comprising:

In response to the absence of the maximum state value, replacing the largest value among all the state values with the maximum state value to generate a state table of the finite state entropy encoding table.
The method according to claim 1, wherein obtaining the state quantity of each character corresponding column in the finite state entropy coding table based on the preset maximum state value and the frequency of occurrence of each character comprises:

The preset maximum state value is multiplied by the frequency of occurrence of each character to respectively obtain the state quantity of the column corresponding to each character in the finite state entropy coding table.
The method according to claim 1, wherein, forming the empty table of the finite state entropy encoding table based on the state quantity of each character corresponding column and the frequency of occurrence of each character includes:

Taking the number of states of each character as the number of spaces in the corresponding column of the finite state entropy coding table, and arranging the order between the columns based on the frequency of occurrence of each character to form the finite state entropy coding An empty list of tables.
The method according to claim 6, wherein arranging the order between the columns based on the frequency of occurrence of each character comprises:

Arrange the corresponding columns from left to right according to the descending order of the occurrence frequency of each character.
The method according to claim 1, wherein obtaining the initial value of each blank based on the number of rows of each blank in the empty table and the frequency of occurrence of the corresponding character comprises:

The ratio of the number of rows where each space is located in the empty table to the frequency of occurrence of the corresponding character is rounded towards zero to obtain the initial value of each space.
The method according to claim 1, wherein traversing the initial table comprises:

The initial table is traversed from left to right and from top to bottom.
The method according to claim 1, wherein m is an integer greater than or equal to 2, and n is an integer greater than or equal to 1.
A generation system of a finite state entropy coding table is characterized in that it comprises:

The frequency of occurrence obtaining module is configured to obtain the frequency of occurrence of each character in the data block based on the proportion of each character in the data block to be encoded in the data block;

Empty table obtaining module, configured to obtain the state quantity of each character corresponding column in the finite state entropy coding table based on the preset maximum state value and the frequency of occurrence of each character, and based on the state quantity of each character corresponding column and the frequency of occurrence of each character forming an empty table of said finite state entropy encoding table;

The initial table obtaining module is configured to obtain the initial value of each blank based on the number of rows of each blank in the empty table and the frequency of occurrence of the corresponding character, and obtain the finite state entropy encoding based on the filling of the initial value of each blank the initial table of tables;

The first judging module is configured to traverse the initial table, and judge whether the initial value of the mth row and the nth column traversed is greater than the initial value of the m-1th row and the nth column;

The second judging module is configured to take the initial value of the mth row and nth column as its temporary value in response to the initial value of the mth row and nth column being greater than the m-1th row and nth column's initial value , and judge whether the temporary value of the mth row and the nth column is repeated with the initial value arranged before it;

A state value module configured to use the temporary value of the mth row and nth column as its state value in response to the fact that the temporary value of the mth row and nth column does not overlap with the initial value arranged before it;

A third judging module, configured to obtain all state values of the finite state entropy coding table in response to the completion of the traversal, and judge whether there is the maximum state value among all the state values; and

A state table generation module, configured to generate a state table of the finite state entropy encoding table based on all state values in response to having the maximum state value.
The system according to claim 11, wherein the state quantity obtained based on the preset maximum state value and the frequency of occurrence of each character in the corresponding column of each character in the finite state entropy coding table includes:

The preset maximum state value is multiplied by the frequency of occurrence of each character to respectively obtain the state quantity of the column corresponding to each character in the finite state entropy coding table.
The system according to claim 11, wherein, forming an empty table of the finite state entropy encoding table based on the state quantity of each character corresponding column and the frequency of occurrence of each character includes:

Taking the number of states of each character as the number of spaces in the corresponding column of the finite state entropy coding table, and arranging the order between the columns based on the frequency of occurrence of each character to form the finite state entropy coding An empty list of tables.
The system according to claim 13, wherein arranging the order of each column based on the frequency of occurrence of each character comprises:

Arrange the corresponding columns from left to right according to the descending order of the occurrence frequency of each character.
The system according to claim 11, wherein the initial value of each blank is obtained based on the number of rows of each blank in the empty table and the frequency of occurrence of the corresponding character:

The ratio of the number of rows where each space is located in the empty table to the frequency of occurrence of the corresponding character is rounded towards zero to obtain the initial value of each space.
The method according to claim 11, wherein m is an integer greater than or equal to 2, and n is an integer greater than or equal to 1.
A computer-readable storage medium storing computer program instructions, the computer program instructions implement the method according to any one of claims 1 to 10 when executed by a processor.
A computer device, comprising a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the method according to any one of claims 1 to 10 is executed.