CN117112601A - Database data compression method, device, equipment and storage medium - Google Patents

Database data compression method, device, equipment and storage medium Download PDF

Info

Publication number
CN117112601A
CN117112601A CN202311099762.7A CN202311099762A CN117112601A CN 117112601 A CN117112601 A CN 117112601A CN 202311099762 A CN202311099762 A CN 202311099762A CN 117112601 A CN117112601 A CN 117112601A
Authority
CN
China
Prior art keywords
compression
partition
database
data
statement
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202311099762.7A
Other languages
Chinese (zh)
Inventor
胡鹰石
付裕
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Jinzhuan Xinke Co Ltd
Original Assignee
Jinzhuan Xinke Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Jinzhuan Xinke Co Ltd filed Critical Jinzhuan Xinke Co Ltd
Priority to CN202311099762.7A priority Critical patent/CN117112601A/en
Publication of CN117112601A publication Critical patent/CN117112601A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/242—Query formulation
    • G06F16/2433—Query languages
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2455—Query execution
    • G06F16/24553—Query execution of query operations

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The application discloses a database data compression method, device, equipment and storage medium, and particularly relates to the technical field of databases. The method comprises the following steps: acquiring a database statement to be processed; determining at least one target partition associated with the database statement to be processed; determining the statement type of the database statement to be processed; and compressing the data to be compressed in the process of executing the database statement to be processed based on the statement type according to the partition compression attribute of each target partition. According to the technical scheme, the data to be compressed is compressed through the partition compression attribute of each target partition, so that the disk resources are saved under the condition that the service performance of the database is not affected.

Description

Database data compression method, device, equipment and storage medium
Technical Field
The embodiment of the application relates to the technical field of computers, in particular to the technical field of databases, and particularly relates to a database data compression method, device, equipment and storage medium.
Background
The partition table in the database classifies and stores the data according to field characteristics and is widely applied to business scenes of large-capacity data.
In practical applications, as the service running time increases, the number of partitions and the data size of the partition table increase. Because the disk resources are limited and the data in the table are important and cannot be cleaned, the disk resource consumption is reduced by the table compression technology.
Currently, the compression method of the database partition table is to compress all partitions of the entire Zhang Fenou table. Because the use frequencies of different partitions are different, the read-write performance of the compressed table is more reduced, so that the table compression technology cannot be applied to the partition table with higher use frequency in the actual service environment.
Disclosure of Invention
The application provides a database data compression method, a device, equipment and a storage medium, which are used for saving disk resources under the condition that the service performance of a database is not affected.
According to an aspect of the present application, there is provided a database data compression method, the method comprising:
acquiring a database statement to be processed;
determining at least one target partition associated with the database statement to be processed;
determining the statement type of the database statement to be processed;
and compressing the data to be compressed in the process of executing the database statement to be processed based on the statement type according to the partition compression attribute of each target partition.
According to another aspect of the present application, there is provided a database data compression apparatus, the apparatus comprising:
the statement acquisition module is used for acquiring a database statement to be processed;
the partition determining module is used for determining at least one target partition associated with the database statement to be processed;
the type determining module is used for determining the statement type of the database statement to be processed;
and the data compression module is used for compressing the data to be compressed in the process of executing the database statement to be processed based on the statement type according to the partition compression attribute of each target partition.
According to another aspect of the present application, a data compression module in a database data compression apparatus includes a first data compression unit; the first data compression unit is used for:
if the statement type is a first statement type and the partition compression attribute exists in the target partition, compressing the data to be compressed in the process of executing the database statement to be processed according to the partition compression attribute of the target partition; the method comprises the steps of,
if the target partition does not have partition compression attributes, determining a target database table to which the target partition belongs, and performing compression processing on data to be compressed in the process of executing the database statement to be processed according to the table compression attributes of the target database table.
According to another aspect of the present application, there is provided an electronic apparatus including:
one or more processors;
a memory for storing one or more programs;
when the one or more programs are executed by the one or more processors, the one or more processors implement any one of the database data compression methods provided by the embodiments of the present application.
According to another aspect of the present application, there is provided a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements any one of the database data compression methods provided by the embodiments of the present application.
The method comprises the steps of obtaining database sentences to be processed; determining at least one target partition associated with the database statement to be processed; determining the statement type of the database statement to be processed; and compressing the data to be compressed in the process of executing the database statement to be processed based on the statement type according to the partition compression attribute of each target partition. According to the technical scheme, the data to be compressed is compressed through the partition compression attribute of each target partition, so that the disk resources are saved under the condition that the service performance of the database is not affected.
Drawings
FIG. 1 is a flow chart of a database data compression method according to a first embodiment of the present application;
FIG. 2 is a flowchart of a database data compression method according to a second embodiment of the present application;
FIG. 3 is a flowchart of a database data compression method according to a third embodiment of the present application;
fig. 4 is a schematic structural diagram of a database data compression device according to a fourth embodiment of the present application;
fig. 5 is a schematic structural diagram of an electronic device implementing a database data compression method according to an embodiment of the present application.
Detailed Description
In order that those skilled in the art will better understand the present application, a technical solution in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present application without making any inventive effort, shall fall within the scope of the present application.
It should be noted that the terms "first," "second," "third," and the like in the description and the claims of the present application and in the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the application described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
In addition, in the technical scheme of the application, the related data such as sentence types, partition compression attributes and the like are collected, stored, used, processed, transmitted, provided, disclosed and the like, which are all in accordance with the regulations of related laws and regulations, and the public welcome is not violated.
Example 1
Fig. 1 is a flowchart of a database data compression method according to a first embodiment of the present application, where the embodiment is applicable to the case of compressing a database partition table, and may be performed by a database data compression device, where the database data compression may be implemented in hardware and/or software, and the database data compression device may be configured in a computer device, for example, a server. As shown in fig. 1, the method includes:
s110, acquiring a database statement to be processed.
Wherein, the database statement to be processed can be a programming language which interacts and operates with the database; the programming language may be at least one of a structured query language (Structured Query Language, SQL) or a hypertext preprocessor (Hypertext Preprocessor, PHP), etc. For example, the database statement to be processed may be a SQL statement to be processed.
S120, determining at least one target partition associated with the database statement to be processed.
The target partition refers to a partition associated with a database statement to be processed in a partition table. For example, a database statement to be processed may be associated with at least one target database table and with at least one target partition under each target database table.
The partition table is a table in which a table is divided into a plurality of partitions according to a certain rule in a database, and each partition corresponds to an independent sub-table. The certain rule may be at least one of a range partition, a list partition, a hash partition, or the like, which is not particularly limited by the embodiment of the present application.
Specifically, at least one target partition associated with the database statement to be processed is determined based on the association relationship between the database statement and the partition. The association relationship between the database statement and the partition can be set manually according to actual conditions or experience values.
For example, statement parsing may be performed on the database statement to be processed, determining a target database table associated with the database statement to be processed, and determining a target partition in the target database table associated with the database statement to be processed.
S130, determining the statement type of the database statement to be processed.
The statement type refers to a type corresponding to the action of the database statement to be processed in the database. Taking SQL language as an example, the corresponding statement types comprise at least one of data operation language types for inquiring and operating data in a database, data control language types for creating, modifying and deleting database objects such as tables and indexes, and the like;
the statement types may include a database statement type that affects the compression attribute of the partition after execution, an SQL statement type that affects the execution result, and other database statement types that do not belong to the former two. For example, the database statement corresponding to the database statement type that affects the partition compression attribute after execution may be a rebuild statement regarding the table; the database statement of the SQL statement type with the compression attribute influencing the execution result can be an SQL statement for performing data migration on the table; other database statements corresponding to database statement types may include positive censored statements of data in the table.
And S140, compressing the data to be compressed in the process of executing the database statement to be processed based on the statement type according to the partition compression attribute of each target partition.
The partition compression attribute is used for representing that the target partition has an independent compression function; the partition compression attribute of the target partition may be a compression algorithm or a compression format, and may specifically be set by a relevant technician according to actual needs, which is not limited in this embodiment.
Specifically, determining partition compression attributes of each target partition, and executing database statements to be processed; and compressing the data to be compressed in the process of executing the database statement to be processed based on the statement type of the database statement to be processed and the partition compression attribute of the target partition.
Further, if the partition compression attribute exists in the target partition, compressing the data to be compressed in the process of executing the database statement to be processed according to the partition compression attribute of the target partition; if the partition compression attribute does not exist in the target partition, determining a target database table to which the target partition belongs, and compressing data to be compressed in the process of executing the database statement to be processed according to the table compression attribute of the target database table.
The table compression attribute of the target database table may be a compression algorithm or a compression format. If each partition in the database table is a partition that is stored independently, the same or different corresponding partition compression attribute may be set for each partition in the database table; if each partition in the database table is not an independent partition, the compressed data corresponding to each partition in the database table is the table compression attribute of the database table.
If each partition in the database table is an independently stored partition, the compression attribute corresponding to each partition may be pre-constructed by a related technician. The method can be specifically set according to the accessed frequency and the data size of the data in the corresponding partition. For example, a compression format or a compression algorithm with higher compression processing efficiency can be set for the compression attribute corresponding to the partition with higher data access frequency, so that the data compression efficiency in the process of executing the database statement is improved.
It can be appreciated that by determining whether the target partition has partition compression attributes, flexibility and adaptability of compression processing of the partition data are improved.
Optionally, after determining the partition compression attribute of the target partition, writing the partition compression attribute into a storage file corresponding to the target partition.
It will be appreciated that the partition compression attribute corresponding to each partition in the partition table may not be readable in different environments, and therefore, the partition compression attribute is stored, and it is determined that the partition compression attribute is still valid after the data environment is replaced.
According to the embodiment of the application, the database statement to be processed is obtained; determining at least one target partition associated with the database statement to be processed; determining the statement type of the database statement to be processed; and compressing the data to be compressed in the process of executing the database statement to be processed based on the statement type according to the partition compression attribute of each target partition. According to the technical scheme, the data to be compressed is compressed through the partition compression attribute of each target partition, so that the disk resources are saved under the condition that the service performance of the database is not affected.
Example two
Fig. 2 is a flowchart of a database data compression method according to a second embodiment of the present application, where the embodiment refines a scheme of "compressing data to be compressed in executing a database statement to be processed based on a statement type according to a partition compression attribute of each target partition" based on the technical scheme of each of the embodiments. It should be noted that, in the embodiments of the present application, parts that are not described in detail may be referred to in the related description of other embodiments. As shown in fig. 2, the method includes:
s210, acquiring a database statement to be processed.
S220, determining at least one target partition associated with the database statement to be processed.
S230, determining the statement type of the database statement to be processed.
S240, if the statement type is the first statement type and the partition compression attribute exists in the target partition, compressing the data to be compressed in the process of executing the database statement to be processed according to the partition compression attribute of the target partition.
The first statement type is used for performing operations such as adding, deleting, checking and the like on data in a partition table in a database to be processed; such as the data control language type used to modify and delete the table. The data to be compressed may be database data that needs to be compressed, which is involved in executing the database statement to be processed.
Specifically, when the statement type of the database statement to be processed is determined to be the first statement type, judging the partition compression attribute of the target partition, and if the partition compression attribute exists in the target partition, proving that the target partition can directly perform partition compression; further, in the process of executing the database statement to be processed, compression processing is performed on the data to be compressed in the process of executing the database statement to be processed based on the partition compression attribute corresponding to the target partition.
S250, if the target partition does not have partition compression attributes, determining a target database table to which the target partition belongs, and compressing data to be compressed in the process of executing the database statement to be processed according to the table compression attributes of the target database table.
It should be noted that, the database tables all correspond to their own table compression attributes, if the partition in the database table has its corresponding partition compression attribute, the partition compression attribute of the partition is higher in priority than the table compression attribute of the affiliated database table; if the partition does not have the corresponding partition compression attribute in the database table, the partition compression attribute of the partition is the table compression attribute of the affiliated database table.
Specifically, if the partition compression attribute does not exist in the target partition, determining a target database table to which the target partition belongs, and after determining the table compression attribute of the target database table, performing compression processing on data to be compressed in the process of executing the database statement to be processed.
According to the embodiment of the application, the database statement to be processed is obtained; determining at least one target partition associated with the database statement to be processed; determining the statement type of the database statement to be processed; if the statement type is the first statement type and the partition compression attribute exists in the target partition, compressing the data to be compressed in the process of executing the database statement to be processed according to the partition compression attribute of the target partition; if the partition compression attribute does not exist in the target partition, determining a target database table to which the target partition belongs, and compressing data to be compressed in the process of executing the database statement to be processed according to the table compression attribute of the target database table. According to the technical scheme, the flexibility and the adaptability of compressing the partition data are improved by judging the priority of the partition compression attribute and the table compression attribute of the database table.
Further, on the basis of the above embodiment, if the statement type is the second statement type, after compressing the data to be compressed in the process of executing the database statement to be processed, pre-compression data before compressing the data to be compressed in the database statement to be processed is obtained; the compressed data after the compression processing of the data to be compressed in the database statement to be processed is obtained; respectively carrying out consistency check on the pre-compression data and the post-compression data which belong to the same database table in the pre-compression data and the post-compression data corresponding to each target partition to obtain a consistency check result; and carrying out correction processing on the compressed data according to the consistency check result to obtain corrected compressed data.
Wherein the second statement type refers to a data statement to be processed for recreating the database table. The pre-compression data refers to data to be compressed. The compressed data refers to data obtained after compression processing is performed on data to be compressed. The consistency check result is a result of consistency check on the data before compression and the data after compression and is used for representing whether the data after compression is correct or not. The corrected compressed data is data after correction processing is performed on the compressed data that has not passed the consistency check.
Specifically, judging the partition compression attribute of the target partition when determining that the statement type of the database statement to be processed is the second statement type, and if the partition compression attribute exists in the target partition; the data to be compressed is compressed by adopting partition compression attribute of the target partition in the process of executing the database statement to be processed; if the partition compression attribute does not exist in the target partition, determining a target database table to which the target partition belongs, and adopting the table compression attribute of the target database table to compress data to be compressed in the process of executing the database statement to be processed.
Whether partition compression or table compression is used, it is necessary to verify the compressed data after the compression process is completed. Further, pre-compression data before compression processing is carried out on the data to be compressed in the database statement to be processed and post-compression data after compression processing is carried out on the data to be compressed in the database statement to be processed are obtained; based on a preset consistency check algorithm, respectively carrying out consistency check on the pre-compression data and the post-compression data which belong to the same database table in the pre-compression data and the post-compression data corresponding to each target partition to obtain a consistency check result; and carrying out correction processing on the compressed data according to the consistency check result to obtain corrected compressed data.
It should be noted that, the preset consistency check algorithm may be at least one of a hash function or a cyclic redundancy check algorithm, which is not limited in particular in the embodiment of the present application.
If the consistency check fails, the data to be compressed, which fails the consistency check, is recompressed by adopting the partition compression attribute corresponding to the target partition or the table compression attribute corresponding to the target database table, so as to obtain corrected compressed data.
It can be understood that after compression processing is performed on the data to be compressed, consistency verification is performed, so that accuracy of the compressed data can be improved.
Example III
Fig. 3 is a flowchart of a database data compression method according to a third embodiment of the present application, where the embodiment refines, based on the technical solutions of the foregoing embodiments, another parallel scheme of "compressing data to be compressed in executing a database statement to be processed based on a statement type according to a partition compression attribute of each target partition". It should be noted that, in the embodiments of the present application, parts that are not described in detail may be referred to in the related description of other embodiments. As shown in fig. 3, the method includes:
S310, acquiring a database statement to be processed.
S320, determining at least one target partition associated with the database statement to be processed.
S330, determining the statement type of the database statement to be processed.
And S340, if the statement type is the third statement type, performing compression attribute verification according to the existence condition of the partition compression attribute of each target partition to obtain a compression attribute verification result.
The third statement type is used for migrating data in the database table and is used for processing database statements. The compression attribute verification result is used for representing whether the compression attribute verification is passed or not.
Specifically, when the type of the database statement to be processed is determined to be the third statement type, based on a preset attribute verification method, performing compression attribute verification on the existence condition of the partition compression attribute of each target partition, and obtaining a compression attribute verification result.
It should be noted that, the preset attribute verification method is set manually according to an actual situation or an empirical value, and may be at least one of regular expression matching or uniqueness checking, which is not limited in particular in the embodiment of the present application.
Optionally, determining a target database table to which each target partition belongs; if the partition compression attribute exists in the target partitions in the same target database table, the partition compression attribute of each target partition in the same target database table is used as the compression attribute to be checked of the same target database table; if the partition compression attribute does not exist in the target partition in the same target database table, the table compression attribute corresponding to the same target database table is used as the compression attribute to be checked of the same target database table; and performing compression attribute verification according to the compression attributes to be verified of each target database table to obtain a compression attribute verification result.
The compressed attribute to be checked refers to the attribute of the target partition in the target database table.
It can be understood that when the partition compression attribute does not exist, the table compression attribute is used as the compression attribute to be checked, so that the compression check can be flexibly adapted to different situations.
Further, performing attribute matching on the compression attributes to be checked among the target database tables, and determining whether the compression attributes to be checked are inconsistent in matching; if yes, the compression attribute verification result is that the verification is not passed; and if not, the compression attribute checking result is that the checking is passed.
Specifically, consistency comparison is carried out on the compression attributes to be checked among the database tables, and whether inconsistent compression attributes to be checked exist among the database tables is determined; if the data exists, the compression modes among the database tables are not uniform, namely the compression attribute verification result is that the verification is failed; if the data is not stored, the compression modes among the database tables are unified, namely the compression attribute verification result is verification passing.
Illustratively, there are two database tables, database table a and database table B, respectively; two partitions exist in the database table A, namely a partition a and a partition b; there are two partitions in database table B, partition c and partition d, respectively. If the partition compression attribute exists between the partition a and the partition B, the compression attribute of the partition a is A, the compression attribute of the partition B is B, the compression attribute of the partition C and the partition D also exist partition compression attributes, the compression attribute of the partition C is C, and the compression attribute of the partition D is D, when the attribute matching is carried out on the compression attributes to be checked of the database table, the partition compression attributes between the partition a and the partition B and the partition C and the partition D are not matched, namely the matching of the attributes to be checked of the database table A and the database table B is inconsistent, and the checking is failed. If the partition a has a partition compression attribute, the partition B has a table compression attribute, the compression attribute of the partition a is A, the compression attribute of the partition B is B, the partition c has a partition compression attribute, the partition d has a table compression attribute, the compression attribute of the partition c is B, the compression attribute of the partition d is A, when the compression attributes of the database tables to be checked are matched, the compression attributes of the partition a and the partition B are matched with the compression attributes of the partition c and the partition d, and the result matching of the database tables A and the database tables B is consistent, and the checking is passed.
It can be understood that in the process of checking the consistency of the compression attribute among the database tables, as long as the partition compression attribute or the table compression attribute among the database tables are consistent, the result of the attribute to be checked can be considered to be matched and consistent, and the checking is passed. If any partition compression attribute or table compression attribute is inconsistent among the database tables, the attribute to be checked is considered to be inconsistent in result matching, and the checking is failed.
S350, determining whether to execute the database statement to be processed according to the compression attribute verification result.
For example, if the compression attribute verification result is that the verification is passed, determining to execute the database statement to be processed; and if the compression attribute checking result is that the checking is not passed, not executing the database statement to be processed.
And S360, if yes, executing the database statement to be processed, and compressing the data to be compressed in the process of executing the database statement to be processed.
According to the embodiment of the application, the database statement to be processed is obtained; determining at least one target partition associated with the database statement to be processed; determining the statement type of the database statement to be processed; if the statement type is the third statement type, before executing the database statement to be processed, performing compression attribute verification according to the existence condition of partition compression attributes of each target partition to obtain a compression attribute verification result; and determining whether to execute the database statement to be processed according to the compression attribute verification result. According to the technical scheme, under the condition that the compression attribute possibly affects the database statement to be processed, the partition attribute among the database tables is checked preferentially, so that smooth execution of the database statement to be processed can be ensured.
Example IV
Fig. 4 is a schematic structural diagram of a database data compression device according to a third embodiment of the present application, which may be suitable for use in compressing a database partition table, where the database data compression device may be implemented in hardware and/or software, and the database data compression may be configured in a computer device, such as a server. As shown in fig. 4, the apparatus includes:
a statement acquiring module 410, configured to acquire a database statement to be processed;
a partition determination module 420, configured to determine at least one target partition associated with the database statement to be processed;
a type determining module 430, configured to determine a statement type of the database statement to be processed;
the data compression module 440 is configured to compress data to be compressed in the process of executing the database statement to be processed based on the statement type according to the partition compression attribute of each target partition.
According to the embodiment of the application, the database statement to be processed is obtained; determining at least one target partition associated with the database statement to be processed; determining the statement type of the database statement to be processed; and compressing the data to be compressed in the process of executing the database statement to be processed based on the statement type according to the partition compression attribute of each target partition. According to the technical scheme, the data to be compressed is compressed through the partition compression attribute of each target partition, so that the disk resources are saved under the condition that the service performance of the database is not affected.
Optionally, the data compression module 440 includes: a first data compression unit; the first data compression unit is used for:
if the statement type is the first statement type and the partition compression attribute exists in the target partition, compressing the data to be compressed in the process of executing the database statement to be processed according to the partition compression attribute of the target partition; the method comprises the steps of,
if the partition compression attribute does not exist in the target partition, determining a target database table to which the target partition belongs, and compressing data to be compressed in the process of executing the database statement to be processed according to the table compression attribute of the target database table.
Optionally, the first data compression unit is further configured to:
if the statement type is the second statement type, acquiring pre-compression data before compression processing is performed on the data to be compressed in the database statement to be processed after the compression processing is performed on the data to be compressed in the process of executing the database statement to be processed; the method comprises the steps of,
acquiring compressed data after compression processing of the data to be compressed in the database statement to be processed;
respectively carrying out consistency check on the pre-compression data and the post-compression data which belong to the same database table in the pre-compression data and the post-compression data corresponding to each target partition to obtain a consistency check result;
And carrying out correction processing on the compressed data according to the consistency check result to obtain corrected compressed data.
Optionally, the data compression module 440 further includes: a second data compression unit; the second data compression unit is used for:
if the statement type is the third statement type, before executing the database statement to be processed, performing compression attribute verification according to the existence condition of partition compression attributes of each target partition to obtain a compression attribute verification result;
and determining whether to execute the database statement to be processed according to the compression attribute verification result.
Optionally, the second data compression unit includes:
a database table determining subunit, configured to determine a target database table to which each target partition belongs;
the attribute determining subunit is used for taking the partition compression attribute of each target partition in the same target database table as the compression attribute to be checked of the same target database table if the partition compression attribute exists in the target partition in the same target database table; if the partition compression attribute does not exist in the target partition in the same target database table, the table compression attribute corresponding to the same target database table is used as the compression attribute to be checked of the same target database table;
And the attribute verification subunit is used for performing compression attribute verification according to the compression attributes to be verified of each target database table to obtain a compression attribute verification result.
Optionally, the attribute verification subunit is specifically configured to:
performing attribute matching on the compression attributes to be checked among the target database tables, and determining whether the compression attributes to be checked with inconsistent matching exist;
if yes, the compression attribute verification result is that the verification is not passed; the method comprises the steps of,
if not, the compression attribute checking result is that the checking is passed.
The database data compression device provided by the embodiment of the application can execute the database data compression method provided by any embodiment of the application, and has the corresponding functional modules and beneficial effects of executing the database data compression methods.
Example five
Fig. 5 is a schematic structural diagram of an electronic device 510 implementing a database data compression method according to an embodiment of the present application. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Electronic equipment may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the applications described and/or claimed herein.
As shown in fig. 5, the electronic device 510 includes at least one processor 511, and a memory communicatively connected to the at least one processor 511, such as a Read Only Memory (ROM) 512, a Random Access Memory (RAM) 513, etc., in which the memory stores computer programs executable by the at least one processor, and the processor 511 may perform various suitable actions and processes according to the computer programs stored in the Read Only Memory (ROM) 512 or the computer programs loaded from the storage unit 518 into the Random Access Memory (RAM) 513. In the RAM513, various programs and data required for the operation of the electronic device 510 can also be stored. The processor 511, the ROM512, and the RAM513 are connected to each other via a bus 515. An input/output (I/O) interface 515 is also connected to bus 515.
Various components in the electronic device 510 are connected to the I/O interface 515, including: an input unit 516 such as a keyboard, a mouse, etc.; an output unit 517 such as various types of displays, speakers, and the like; a storage unit 518 such as a magnetic disk, optical disk, etc.; and a communication unit 519 such as a network card, modem, wireless communication transceiver, or the like. The communication unit 519 allows the electronic device 510 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunications networks.
The processor 511 may be a variety of general and/or special purpose processing components with processing and computing capabilities. Some examples of processor 511 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, digital Signal Processors (DSPs), and any suitable processor, controller, microcontroller, etc. The processor 511 performs the various methods and processes described above, such as database data compression methods.
In some embodiments, the database data compression method may be implemented as a computer program tangibly embodied on a computer-readable storage medium, such as storage unit 518. In some embodiments, some or all of the computer program may be loaded and/or installed onto the electronic device 510 via the ROM512 and/or the communication unit 519. When the computer program is loaded into RAM513 and executed by processor 511, one or more steps of the database data compression method described above may be performed. Alternatively, in other embodiments, processor 511 may be configured as a database data compression method in any other suitable manner (e.g., by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.
A computer program for carrying out methods of the present application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be implemented. The computer program may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of the present application, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which a user can provide input to the electronic device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), blockchain networks, and the internet.
The computing system may include clients and servers. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical hosts and VPS service are overcome.
It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps described in the present application may be performed in parallel, sequentially, or in a different order, so long as the desired results of the technical solution of the present application are achieved, and the present application is not limited herein.
The above embodiments do not limit the scope of the present application. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the scope of the present application.

Claims (10)

1. A method of database data compression, comprising:
acquiring a database statement to be processed;
determining at least one target partition associated with the database statement to be processed;
determining the statement type of the database statement to be processed;
and compressing the data to be compressed in the process of executing the database statement to be processed based on the statement type according to the partition compression attribute of each target partition.
2. The method according to claim 1, wherein the compressing the data to be compressed in executing the database statement to be processed based on the statement type according to the partition compression attribute of each of the target partitions includes:
if the statement type is a first statement type and the partition compression attribute exists in the target partition, compressing the data to be compressed in the process of executing the database statement to be processed according to the partition compression attribute of the target partition; the method comprises the steps of,
if the target partition does not have partition compression attributes, determining a target database table to which the target partition belongs, and performing compression processing on data to be compressed in the process of executing the database statement to be processed according to the table compression attributes of the target database table.
3. The method according to claim 2, wherein the method further comprises:
if the statement type is the second statement type, after the compressing process is performed on the data to be compressed in the process of executing the database statement to be processed, the method further includes:
acquiring pre-compression data before compression processing of the data to be compressed in the database statement to be processed; the method comprises the steps of,
Acquiring compressed data after compression processing of the data to be compressed in the database statement to be processed;
respectively carrying out consistency check on the pre-compression data and the post-compression data which belong to the same database table in the pre-compression data and the post-compression data corresponding to each target partition to obtain a consistency check result;
and carrying out correction processing on the compressed data according to the consistency check result to obtain corrected compressed data.
4. The method according to claim 1, wherein the compressing the data to be compressed in executing the database statement to be processed based on the statement type according to the partition compression attribute of each of the target partitions includes:
if the statement type is a third statement type, before executing the database statement to be processed, performing compression attribute verification according to the existence condition of partition compression attributes of each target partition to obtain a compression attribute verification result;
and determining whether to execute the database statement to be processed according to the compression attribute verification result.
5. The method according to claim 4, wherein the performing the compression attribute verification according to the existence of the partition compression attribute of each target partition to obtain the compression attribute verification result includes:
Determining a target database table to which each target partition belongs;
if the partition compression attribute exists in the target partitions in the same target database table, the partition compression attribute of each target partition in the same target database table is used as the compression attribute to be checked of the same target database table;
if the partition compression attribute does not exist in the target partition in the same target database table, the table compression attribute corresponding to the same target database table is used as the compression attribute to be checked of the same target database table;
and performing compression attribute verification according to the compression attributes to be verified of each target database table to obtain a compression attribute verification result.
6. The method according to claim 5, wherein the performing the compression attribute verification according to the compression attribute to be verified of each target database table to obtain the compression attribute verification result includes:
performing attribute matching on the compression attributes to be checked among the target database tables, and determining whether the compression attributes to be checked with inconsistent matching exist;
if yes, the compression attribute verification result is that the verification is not passed; the method comprises the steps of,
if not, the compression attribute verification result is that the verification passes.
7. A database data compression apparatus, comprising:
the statement acquisition module is used for acquiring a database statement to be processed;
the partition determining module is used for determining at least one target partition associated with the database statement to be processed;
the type determining module is used for determining the statement type of the database statement to be processed;
and the data compression module is used for compressing the data to be compressed in the process of executing the database statement to be processed based on the statement type according to the partition compression attribute of each target partition.
8. The apparatus of claim 7, wherein the data compression module comprises a first data compression unit; the first data compression unit is used for:
if the statement type is a first statement type and the partition compression attribute exists in the target partition, compressing the data to be compressed in the process of executing the database statement to be processed according to the partition compression attribute of the target partition; the method comprises the steps of,
if the target partition does not have partition compression attributes, determining a target database table to which the target partition belongs, and performing compression processing on data to be compressed in the process of executing the database statement to be processed according to the table compression attributes of the target database table.
9. An electronic device, comprising:
one or more processors;
a memory for storing one or more programs;
the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the database data compression method of any of claims 1-6.
10. A computer readable storage medium having stored thereon a computer program, which when executed by a processor implements a database data compression method according to any of claims 1-6.
CN202311099762.7A 2023-08-29 2023-08-29 Database data compression method, device, equipment and storage medium Pending CN117112601A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311099762.7A CN117112601A (en) 2023-08-29 2023-08-29 Database data compression method, device, equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311099762.7A CN117112601A (en) 2023-08-29 2023-08-29 Database data compression method, device, equipment and storage medium

Publications (1)

Publication Number Publication Date
CN117112601A true CN117112601A (en) 2023-11-24

Family

ID=88796168

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311099762.7A Pending CN117112601A (en) 2023-08-29 2023-08-29 Database data compression method, device, equipment and storage medium

Country Status (1)

Country Link
CN (1) CN117112601A (en)

Similar Documents

Publication Publication Date Title
CN112528067A (en) Graph database storage method, graph database reading method, graph database storage device, graph database reading device and graph database reading equipment
CN113220710B (en) Data query method, device, electronic equipment and storage medium
CN114816578A (en) Method, device and equipment for generating program configuration file based on configuration table
CN112948081B (en) Method, device, equipment and storage medium for processing tasks in delayed mode
CN116226150A (en) Data processing method, device, equipment and medium based on distributed database
CN116185389A (en) Code generation method and device, electronic equipment and medium
CN117112601A (en) Database data compression method, device, equipment and storage medium
CN115904240A (en) Data processing method and device, electronic equipment and storage medium
CN115639966A (en) Data writing method and device, terminal equipment and storage medium
CN112887426B (en) Information stream pushing method and device, electronic equipment and storage medium
CN115168358A (en) Database access method and device, electronic equipment and storage medium
CN114969444A (en) Data processing method and device, electronic equipment and storage medium
US20230132173A1 (en) Data reading method, device and storage medium
CN115757928A (en) Data query method and device, electronic equipment and storage medium
CN118012897A (en) Heterogeneous database grammar conversion method, device, equipment and storage medium
CN115203246A (en) Linked list query method and device, electronic equipment and storage medium
CN115421665A (en) Data storage method, device, equipment and storage medium
CN115794830A (en) Data value determination method and device, electronic equipment and storage medium
CN117407044A (en) Configuration method, device, equipment and medium of cross-terminal assembly
CN114416881A (en) Real-time synchronization method, device, equipment and medium for multi-source data
CN112948246A (en) AB test control method, device, equipment and storage medium of data platform
CN117573677A (en) Data updating method, device, equipment and medium based on transverse table database structure
CN115525659A (en) Data query method and device, electronic equipment and storage medium
CN117278648A (en) Protocol dynamic adaptation method, device, electronic equipment and storage medium
CN117194435A (en) Index data updating method, device, equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination