WO2015139193A1

WO2015139193A1 - Method and apparatus for conversion of data storage formats

Info

Publication number: WO2015139193A1
Application number: PCT/CN2014/073576
Authority: WO
Inventors: 李怀洲; 姜旭栋
Original assignee: 华为技术有限公司
Priority date: 2014-03-18
Filing date: 2014-03-18
Publication date: 2015-09-24
Also published as: CN105378716B; CN105378716A

Abstract

Provided in the embodiment are a method and apparatus for the conversion of data storage formats, which relate to the technical field of database systems and can enable a dynamic determination of the low-level storage format by a database system so as to realize the automatic regulation and optimization functions of the system. The method comprises: if the system performance indexes of a database in which data is stored in a first storage format meet a conversion condition set by a user, then determining a second storage form required for data storage in the database; converting the storage format of the data from the first storage format to the second storage format; judging whether the compression ratio of the database undergoing storage format conversion meets a first preset condition, sorting the data in the database, and testing whether the time for sorting meets a second preset condition; if the compression ratio meets the first preset condition and the time for sorting meets the second preset condition, then providing external services; or if the compression ratio fails to meet the first preset condition and/or the time for sorting fails to meet the second preset condition, then re-determining the second storage format.

Description

Method and device for converting data storage format

Technical field

The present invention relates to the field of database system technologies, and in particular, to a method and an apparatus for converting a data storage format.

Background technique

With the deepening of the degree of social informatization, the use of database systems has become more widespread, and the ever-increasing massive data and ever-increasing data expansion have placed new demands on database systems.

The data stored in the database has a certain storage format, and different storage formats can affect the performance of the database system. The row storage structure can quickly complete data loading, and the adaptation to dynamic load is high, but the row storage structure cannot support fast query processing, and the space utilization rate is not easily improved. Although a better compression ratio is achieved by entropy coding and column correlation performance, complex data storage implementations result in increased decompression overhead. The column storage structure distributes the different domains of the same record and the reconstruction of these records will bring a large overhead, but the column storage can avoid reading unnecessary columns, and compressing similar data in one column can achieve higher Compression ratio.

At present, the advantages and disadvantages of comprehensive row storage and column storage have resulted in a variety of row and column combination storage methods, 3⁄4 port PAX or row 歹 J J J J J J J Record Record Record Record Record Record Record Record Record Record Record Record

However, the existing database's row, column, or row-column mixed storage format is completely dependent on the initial setup of the database initialization, which is the underlying storage format of the database that the user specified when creating the database. When the user needs to change the database storage format, the database administrator (DBA, Database Administrator) is required to manually modify it offline. Summary of the invention

The embodiment of the invention provides a method and a device for converting a data storage format, which can enable the database system to dynamically determine the underlying storage format of the database according to the load situation, realize the automatic adjustment and optimization function of the system, reduce the throughput of the query statement, and improve the system. Storage space and utilization.

In order to achieve the above object, embodiments of the present invention use the following technical solutions: In a first aspect, an embodiment of the present invention provides a controller, including:

a decision unit, configured to: if the system of the database storing the data in the first storage format satisfies a conversion condition set by the user, determine a second storage format required for storing the data in the database;

a storage format conversion unit, configured to convert a storage format of data in the database from the first storage format to the second storage format;

a feedback unit, configured to determine whether the compression ratio of the database after the storage format conversion satisfies the first preset condition, and sort the data in the database after the storage format conversion, and test whether the sorting time meets the second preset condition; If the compression ratio satisfies the first preset condition, and the sorting time satisfies the second preset condition, performing external service, or if the compression ratio does not satisfy the first preset condition, and/or the sorting If the time does not satisfy the second preset condition, the feedback information is sent to the decision unit, so that the decision unit needs to re-determine the data stored in the database according to the core indicator threshold to be set by the user in the feedback information. The second storage format.

In a first possible implementation manner of the first aspect, the method further includes: the determining unit, configured to determine the system performance indicator before determining a second storage format required for storing data in the database Whether the core indicator threshold set by the user is met; if the system performance indicator meets the core indicator threshold, it is determined whether the system performance indicator satisfies the conversion condition.

In conjunction with the first possible implementation of the first aspect, in a second possible implementation manner of the first aspect, the determining unit is further configured to: if the system performance indicator does not meet the conversion condition, The format of the stored data in the database is the first storage format.

In combination with the first aspect of the foregoing or the first possible implementation of the first aspect to any one of the possible implementations of the second possible implementation of the first aspect, the third possible implementation of the first aspect In the mode, the controller further includes a data collection unit, and the data collection unit is configured to collect the system performance indicator.

In combination with the first aspect of the foregoing or the first possible implementation of the first aspect to any one of the third possible implementations of the first aspect, at the first In a fourth possible implementation manner of the aspect, the determining unit is further configured to: after determining a second storage format required for storing data in the database, determine, according to user configuration information, data in the database. The storage format is converted from the first storage format to the conversion timing of the second storage format.

With reference to the fourth possible implementation of the first aspect, in a fifth possible implementation manner of the first aspect, the storage format conversion unit is specifically configured to use, according to the second storage format and the conversion time, The data in the database is reorganized in the buffer, and if the amount of data in the buffer reaches the disk write threshold, the data in the buffer is written to the disk.

In conjunction with the fifth possible implementation of the first aspect, in a sixth possible implementation manner of the first aspect, the storage format conversion unit is specifically configured to: if the second part of data in the buffer The storage format is a single column storage, and the storage format of the data in the buffer is converted from the first storage format to the single column storage, and the data in the buffer is compressed and stored; or The second storage format of the data in the buffer is a row-column mixed storage or a row storage, and the storage format of the data in the buffer is converted from the first storage format to the row-column mixed storage or The row stores and stores data in the buffer.

In conjunction with the fifth possible implementation of the first aspect, in a seventh possible implementation manner of the first aspect, the controller further includes a reading unit;

The reading unit is configured to: before the storage format conversion unit reorganizes data in the database in a buffer according to the second storage format and the conversion time, if the first storage format For row storage, the data in the database is read in rows.

In conjunction with the first aspect of the foregoing or the first possible implementation of the first aspect to any one of the seventh possible implementations of the first aspect, in an eighth possible implementation of the first aspect,

The data collection unit is further configured to: after the compression ratio meets the first preset condition, and the sorting time satisfies the second preset condition, after performing the external service, collecting the storage format converted The data throughput of the database and the response time of the query statement. In combination with the foregoing first aspect or the first possible implementation of the first aspect to any one of the eighth possible implementation manners of the first aspect, in a ninth possible implementation manner of the first aspect, The system performance indicators include at least the amount of data, the average amount of data accessed by the query, the number of rows processed, the ratio of rows read, the column ratio of the average query, and the proportion of query statements.

In a second aspect, an embodiment of the present invention provides a controller, including:

a processor, configured to determine a second storage format required to store data in the database if a system performance indicator of a database storing data in a first storage format satisfies a conversion condition set by a user; After the storage format of the data in the database is converted from the first storage format to the second storage format, determining whether the compression ratio of the database after the storage format conversion satisfies the first preset condition, and converting the storage format The data in the database is sorted, and the test sorting time meets the second preset condition; if the compression ratio satisfies the first preset condition, and the sorting time satisfies the second preset condition, the external service is performed, or If the compression ratio does not satisfy the first preset condition, and/or the sorting time does not satisfy the second preset condition, the first required for storing data in the database is re-determined according to the core indicator threshold to be set by the user. Two storage formats;

a format converter for converting a storage format of data in the database from the first storage format to the second storage format determined by the processor.

In a first possible implementation manner of the second aspect, the processor is further configured to determine whether the system performance indicator meets a user setting before determining a second storage format required for storing data in the database The core indicator threshold; if the system performance indicator meets the core indicator threshold, it is determined whether the system performance indicator satisfies the conversion condition.

With reference to the first possible implementation of the second aspect, in a second possible implementation manner of the second aspect, the processor is further configured to: if the system performance indicator does not meet the conversion condition, The format of the data stored in the database is the first storage format.

Combining the foregoing second aspect or the first possible implementation of the second aspect to the The second possible implementation manner of the second aspect, the third possible implementation manner of the second aspect, the controller further includes a data collector;

The data collector is configured to collect the system performance indicators.

In combination with the foregoing second aspect or the first possible implementation of the second aspect to any one of the third possible implementations of the second aspect, the fourth possible implementation of the second aspect In the mode, the processor is further configured to: after determining the second storage format required for storing the data in the database, determine, according to the user configuration information, a storage format of the data in the database from the first storage. The format is converted to the conversion time of the second storage format.

With the fourth possible implementation of the second aspect, in a fifth possible implementation manner of the second aspect, the format converter is specifically configured to use, according to the second storage format and the conversion time, The data in the database is reorganized in the buffer, and if the amount of data in the buffer reaches the disk write threshold, the data in the buffer is written to the disk.

With reference to the fifth possible implementation of the second aspect, in a sixth possible implementation manner of the second aspect, the format converter is specifically configured to: if the second of the data in the buffer The storage format is a single column storage, and the storage format of the data in the buffer is converted from the first storage format to the single column storage, and the data in the buffer is compressed and stored; or The second storage format of the data in the buffer is a row-column mixed storage or a row storage, and the storage format of the data in the buffer is converted from the first storage format to the row-column mixed storage or The row stores and stores data in the buffer.

With reference to the fifth possible implementation of the second aspect, in a seventh possible implementation manner of the second aspect, the processor is further configured to perform, according to the second storage format, The conversion time, before the data in the database is recombined in the buffer, if the first storage format is row storage, the data in the database is read in rows.

In combination with the foregoing second aspect or the first possible implementation of the second aspect to any one of the seventh possible implementations of the second aspect, in a second aspect In the eighth possible implementation,

The data collector is further configured to: after the compression ratio meets the first preset condition, and the sorting time satisfies the second preset condition, after performing the external service, collecting the storage format converted The data throughput of the database and the response time of the query statement.

In combination with the foregoing second aspect or the first possible implementation of the second aspect to any one of the eighth possible implementations of the second aspect, the ninth possible implementation of the second aspect The system performance indicator includes at least a data volume, a query average access data amount, a processed row number, a read row number ratio, a query average access column ratio, and a proportion of the query statement.

In a third aspect, an embodiment of the present invention provides a data storage format conversion method, including:

Step a: determining, if the system performance indicator of the database storing the data in the first storage format satisfies a conversion condition set by the user, determining a second storage format required for storing the data in the database;

Step b: converting a storage format of the data in the database from the first storage format to the second storage format;

Step c: determining whether the compression ratio of the database after the storage format conversion satisfies the first preset condition, and sorting the data in the database after the storage format conversion, and testing whether the sorting time meets the second preset condition;

Step d: if the compression ratio satisfies the first preset condition, and the sorting time satisfies the second preset condition, performing external service; or, if the compression ratio does not satisfy the first preset condition, and/or If the sorting time does not satisfy the second preset condition, the foregoing step is re-executed according to the core '^ index threshold to be set by the user.

In a first possible implementation manner of the third aspect, before the determining the second storage format required for storing the data in the database, the method further includes:

Determining whether the system performance indicator meets a core indicator threshold set by the user; if the system performance indicator meets the core indicator threshold, determining whether the system performance indicator satisfies the conversion condition.

In combination with the first possible implementation of the third aspect, the second aspect of the third aspect In a possible implementation manner, if the system performance indicator does not meet the conversion condition, maintaining a format of the stored data in the database is the first storage format.

With reference to the foregoing third aspect, or any one of the first possible implementation manner of the third aspect to the second possible implementation manner, in a third possible implementation manner of the third aspect, the method Also includes:

Collect the system performance indicators.

In combination with the third aspect of the foregoing or the first possible implementation of the third aspect to any one of the third possible implementation manners, in the fourth possible implementation manner of the third aspect, After the second storage format required for storing data in the database, the method further includes:

And determining, according to the user configuration information, a conversion timing of converting a storage format of the data in the database from the first storage format to the second storage format.

In conjunction with the fourth possible implementation of the third aspect, in a fifth possible implementation manner of the third aspect, the converting a storage format of the data in the database from the first storage format to the The second storage format specifically includes:

Reconstructing data in the database in a buffer according to the second storage format and the conversion time;

If the amount of data in the buffer reaches the disk write threshold, the data in the buffer is written to disk.

With the fifth possible implementation of the third aspect, in a sixth possible implementation manner of the third aspect, the data in the buffer is written to the disk, and the method includes:

If the second storage format of the data in the buffer is a single column storage, converting a storage format of the data in the buffer from the first storage format to the single column storage, and compressing and Storing data in the buffer; or

If the second storage format of the data in the buffer is row-column mixed storage or row storage, converting a storage format of data in the buffer from the first storage format to the row-column mix The storage or the row is stored and the data in the buffer is stored. With reference to the fifth possible implementation manner of the third aspect, in a seventh possible implementation manner of the third aspect, the performing, according to the second storage format and the switching moment, in a buffer Before the data is reorganized in the database, the method further includes: if the first storage format is row storage, reading the data in the database by row.

With reference to the foregoing third aspect, or any one of the first possible implementation manner of the third aspect to the seventh possible implementation manner, in an eighth possible implementation manner of the third aspect, If the compression ratio meets the first preset condition, and the sorting time meets the second preset condition, after performing the external service, the method further includes: collecting data throughput of the database after the storage format conversion And query statement response time.

With reference to the foregoing third aspect, or any one of the first possible implementation to the eighth possible implementation of the third aspect, in a ninth possible implementation manner of the third aspect, the system The performance indicators include at least the amount of data, the average amount of data accessed by the query, the number of rows processed, the ratio of rows read, the column ratio of the average query, and the proportion of query statements.

Embodiments of the present invention provide a data storage format conversion method and apparatus. If a system performance indicator of a database storing data in a first storage format satisfies a conversion condition set by a user, the controller determines a required number of data stored in the database. The second storage format converts the storage format of the data in the database from the first storage format to the second storage format, and then the controller determines whether the compression ratio of the database after the storage format conversion satisfies the first preset condition, and stores The data in the formatted database is sorted, and the test sorting time meets the second preset condition. If the compression ratio satisfies the first preset condition, and the sorting time satisfies the second preset condition, the external service is performed; or, if If the compression ratio does not satisfy the first preset condition, and/or the sorting time does not satisfy the second preset condition, the second storage format required to determine the data stored in the database is re-executed according to the core indicator threshold to be set by the user. Through the scheme, the controller continuously determines the optimal storage format of the data in the database system by monitoring the actual running data of the system, that is, the controller can enable the database system to dynamically determine the storage format of the data in the database system according to the load situation. It solves the problem that the database administrator needs to manually modify the database storage format, and the system storage space and utilization rate are low. The self-decision data storage format reduces the throughput of the query statement and improves the storage space of the system. Utilization rate.

DRAWINGS

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the description of the prior art will be briefly described below. Obviously, the drawings in the following description are only It is a certain embodiment of the present invention, and other drawings can be obtained from those skilled in the art without any creative work.

1 is a schematic structural view 1 of a controller according to an embodiment of the present invention;

2 is a schematic structural diagram 2 of a controller according to an embodiment of the present invention;

3 is a schematic structural diagram 3 of a controller according to an embodiment of the present invention;

4 is a schematic structural diagram 4 of a controller according to an embodiment of the present invention;

FIG. 5 is a schematic flowchart of a method for converting a data storage format according to an embodiment of the present invention. FIG. 6 is a schematic flowchart of a method for converting a data storage format according to an embodiment of the present invention.

The technical solutions in the embodiments of the present invention are clearly and completely described in the following with reference to the accompanying drawings in the embodiments of the present invention. It is obvious that the described embodiments are only a part of the embodiments of the present invention, but not all embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present invention without creative efforts are within the scope of the present invention.

The various techniques described in this paper apply to the database domain, such as: dynamic optimization of database underlying data storage formats, data distribution in database clusters, database materialization strategies, database indexing strategies, and so on.

Currently, database-based applications are mainly divided into two types: OLTP (On-Line Transaction Processing) and ONAP Analytical Processing (OLAP). The former needs to be processed frequently to "write". The transactional query of operations, the latter focuses on processing analytical queries involving a large number of "read" operations. Column storage has a big advantage in read operations, and is very suitable for 0LAP queries, but support for write operations is not ideal, so it is not suitable for 0LTP queries. Row storage is very good for 0LTP query support.

The advantage of the row storage structure is the high adaptability of fast data loading and dynamic loading, because row storage guarantees that all domains of the same record are on the same node. However, the disadvantages of row storage are obvious, for example, it does not support fast query processing, because when a query is only for a few columns in a multi-list, it cannot skip unnecessary column reads; Columns of data values, row storage is not easy to obtain a very high compression ratio, that is, space utilization is not easy to increase significantly. Although a better compression ratio can be obtained by entropy coding and column correlation, complex data storage implementations result in increased decompression overhead.

Column storage scatters the different domains of the same record, and the refactoring of these records introduces significant overhead. However, column storage can avoid reading unnecessary columns, and compressing similar data in a column can achieve a higher compression ratio.

At present, the advantages and disadvantages of comprehensive row and column storage have resulted in a variety of row and column combination storage methods. For example, PAX or RCFile storage methods optimize system performance by optimizing the underlying storage format. However, the existing database's row, column, or row-column mixed storage format is completely dependent on the initial setup of the database when it is initialized, which is the underlying storage format of the database that the user specified when creating the database. When the user needs to change the database storage format, the database administrator (DBA, Database Administrator) needs manual offline modification, and the system automatic adjustment optimization function is missing.

The invention provides a data storage format conversion method and device, which can enable the database system to dynamically determine the underlying storage format of the database according to the load situation, realize the automatic adjustment and optimization function of the system, reduce the throughput of the query statement, and improve the storage space of the system. And utilization. Embodiment 1

An embodiment of the present invention provides a controller, as shown in FIG. 1, including: The determining unit 10 is configured to determine, if the system performance indicator of the database storing the data in the first storage format meets a conversion condition set by the user, determining a second storage format required for storing the data in the database;

a storage format conversion unit 11 for converting a storage format of data in the database from the first storage format to the second storage format determined by the decision unit 10;

The feedback unit 12 is configured to determine whether the compression ratio of the database after the storage format conversion satisfies the first preset condition, and sort the data in the database after the storage format conversion, and test whether the sorting time meets the second preset condition. And if the compression ratio satisfies the first preset condition, and the sorting time satisfies the second preset condition, performing external service, or if the compression ratio does not satisfy the first preset condition, and/or If the sorting time does not satisfy the second preset condition, the feedback information is sent to the decision unit 10, so that the determining unit re-determines the stored data in the database according to the core indicator threshold to be set by the user in the feedback information. The second storage format required.

Further, the determining unit 10 is further configured to: before determining a second storage format required for storing data in the database, determining whether the system performance indicator meets a core indicator threshold set by a user; If the indicator meets the core indicator threshold, it is determined whether the system performance indicator satisfies the conversion condition.

Further, the determining unit 10 is further configured to: if the system performance indicator does not satisfy the conversion condition, maintain a format of the stored data in the database as the first storage format.

Further, as shown in FIG. 2, the controller further includes a data collection unit 13; the data collection unit 13 is further configured to collect the system performance indicator.

Further, the determining unit 10 is further configured to: after determining the second storage format required for storing the data in the database, determine, according to the user configuration information, a storage format of the data in the database from the first The storage format is converted to a conversion time of the second storage format.

Further, the storage format conversion unit 11 is specifically configured to: in the buffer according to the second storage format and the conversion time determined by the decision unit 10 Data reorganization in the database, if the amount of data in the buffer reaches the disk write threshold, the data in the buffer is written to the disk.

Further, the storage format conversion unit 1 is specifically configured to: if the second storage format of the data in the buffer is a single column storage, store the data in the buffer from a storage format Converting the first storage format to the single column storage, and compressing and storing the data in the buffer; or if the second storage format of the data in the buffer is a row or column hybrid storage or a row storage And converting the storage format of the data in the buffer from the first storage format to the row-column mixed storage or the row storage, and storing the data in the buffer.

Further, as shown in FIG. 2, the controller further includes a reading unit 1 4 for the storage format conversion unit 1 1 according to the second storage format and the At the time of conversion, before the data in the database is recombined in the buffer, if the first storage format is row storage, the data in the database is read in rows.

Further, the data collection unit 13 is further configured to: after the external compression is performed, if the compression ratio satisfies the first preset condition, and the scheduling time meets the second preset condition, The data throughput of the database after the format conversion and the response time of the query statement.

Further, the system performance indicator includes at least a data volume, a query average access data amount, a processing row number as a read row number ratio, a query average access column ratio, and a proportion of the query statement.

The invention provides a controller, which mainly comprises a decision unit, a storage format conversion unit and a feedback unit. If the system performance indicator of the database storing the data in the first storage format satisfies the conversion condition set by the user, the decision unit determines a second storage format required for storing the data in the database, and then the storage format conversion unit sets the data in the database. The storage format is converted from the first storage format to the second storage format. Finally, the feedback unit determines whether the compression ratio of the database after the storage format conversion satisfies the first preset condition, and sorts the data in the database after the storage format conversion. Whether the test sorting time satisfies the second preset condition, if the compression ratio satisfies the first preset condition, and the sorting time is satisfied If the second predetermined condition is used, the external service is performed; or if the compression ratio does not satisfy the first preset condition, and/or the sorting time does not satisfy the second preset condition, the feedback information is sent to the decision unit, so that the The core indicator threshold to be set by the user in the feedback information re-determines the second storage format required to store the data in the database. Through the scheme, the controller continuously determines the optimal storage format of the data in the database system by monitoring the actual running data of the system, that is, the controller can enable the database system to dynamically determine the storage format of the data in the database system according to the load situation, and solve the current When changing the database storage format, the database administrator needs to manually modify the offline, the system storage space and low utilization rate, through the self-decision data storage format, reduce the throughput of the query statement, and at the same time improve the storage space and utilization of the system. Embodiment 2

The embodiment of the present invention provides a controller, as shown in FIG. 3, including: a processor 20, configured to: if a system performance indicator of a database storing data in a first storage format satisfies a conversion condition set by a user, Determining a second storage format required for storing data in the database; determining, after the format converter 21 converts a storage format of the data in the database from the first storage format to the second storage format Whether the compression ratio of the formatted database satisfies the first preset condition, and sorts the data in the database after the storage format conversion, and tests whether the sorting time satisfies the second preset condition; if the compression ratio satisfies the first a preset condition, and if the sorting time satisfies the second preset condition, the external service is performed, or if the compression ratio does not satisfy the first preset condition, and/or the sorting time does not satisfy the second preset Condition, the second storage format required for storing data in the database is re-determined according to a core indicator threshold to be set by the user;

The format converter 2 1 is configured to convert a storage format of data in the database from the first storage format to the second storage format determined by the processor 20.

Further, the processor 20 is further configured to: before determining a second storage format required for storing data in the database, determining whether the system performance indicator meets a core indicator threshold set by a user; The performance indicator meets the core indicator threshold. Then, it is determined whether the system performance indicator satisfies the conversion condition.

Further, the processor 20 is further configured to keep the format of the stored data in the database as the first storage format if the system performance indicator does not satisfy the conversion condition.

Further, as shown in FIG. 4, the controller further includes a data collector 22, and the data collector 22 is configured to collect the system performance indicators.

Further, the processor 20 is further configured to: after determining a second storage format required for storing data in the database, determine, according to user configuration information, a storage format of data in the database from the first A storage format is converted to a conversion instant of the second storage format.

Further, the format converter 2 1 is specifically configured to reassemble data in the database in a buffer according to the second storage format and the conversion time, if data in the buffer When the amount reaches the disk write threshold, the data in the buffer is written to the disk.

Further, the format converter 2 1 is specifically configured to: if the second storage format of the data in the buffer is a single column storage, store a format of the data in the buffer from the Converting the first storage format to the single column storage, and compressing and storing the data in the buffer; or, if the second storage format of the data in the buffer is a row or column hybrid storage or a row storage, And storing the storage format of the data in the buffer from the first storage format to the row and column hybrid storage or the row storage, and storing the data in the buffer.

Further, the processor 20 is further configured to: before the storage format converter reorganizes data in the database in a buffer according to the second storage format and the conversion moment, The first storage format is row storage, and the data in the database is read in rows.

Further, the data collector 2 2 is configured to collect the storage after the external service is performed if the compression ratio satisfies the first preset condition and the sorting time satisfies the second preset condition. The data throughput of the formatted database and the response time of the query statement. Further, the system performance indicator includes at least a data volume, a query average access data amount, a processed row number as a read row number ratio, a query average access column ratio, and a proportion of the query statement.

Embodiments of the present invention provide a controller, which mainly includes a processor and a format converter. If the system performance indicator of the database storing the data in the first storage format satisfies the conversion condition set by the user, the controller determines a second storage format required for storing the data in the database, and stores the data in the database from the first format. The storage format is converted into the second storage format, and then the controller determines whether the compression ratio of the database after the storage format conversion satisfies the first preset condition, and sorts the data in the database after the storage format conversion, and tests whether the sorting time is satisfied. a second preset condition, if the compression ratio satisfies the first preset condition, and the sorting time satisfies the second preset condition, the external service is performed; or, if the compression ratio does not satisfy the first preset condition, and/or the sorting time is not When the second preset condition is met, the second storage format required to determine the data stored in the database is re-executed according to the core indicator threshold to be set by the user. Through the scheme, the controller continuously determines the optimal storage format of the data in the database system by monitoring the actual running data of the system, that is, the controller can enable the database system to dynamically determine the storage format of the data in the database system according to the load situation, and solve the current When changing the database storage format, the database administrator needs to manually modify the offline, the system storage space and low utilization rate, through the self-decision data storage format, reduce the throughput of the query statement, and at the same time improve the storage space and utilization of the system. Embodiment 3

An embodiment of the present invention provides a data storage format conversion method. As shown in FIG. 5, the method includes:

S 1 0 1. If the system performance indicator of the database storing the data in the first storage format satisfies the conversion condition set by the user, the controller determines the second storage format required for storing the data in the database.

The existing database's row, column, or row and column hybrid storage format is completely dependent on the initial setting of the database initialization, that is, the underlying storage format of the database specified by the user when creating the database. DB A hand is required when the user needs to change the database storage format Dynamic offline modification, lack of automatic adjustment and optimization function.

In order to solve the problem that the system cannot automatically adjust the storage format of the data in the optimized database, the present invention provides a data storage format conversion method, which can enable the database system to dynamically determine the underlying storage format of the database according to the load situation, and realize the automatic adjustment and optimization function of the system. .

In practical applications, the 0L TP application and the 0LAP application of the database have advantages in write operations and read operations, respectively. In order to integrate the advantages and disadvantages of row storage and column storage, various types of 'J combination storage methods, 0LTP and 0LAP bugs are generated. In an application environment for 0LTP and 0LAP convergence, database initialization exists in rows.

To complete the conversion of the data storage format in the database system, the controller needs to first determine the storage format required to store the data in the database, that is, to determine which format the data stored in the database needs to be stored.

Specifically, if the system performance indicator of the database storing the data in the first storage format satisfies the conversion condition set by the user, the controller determines a second storage format required for storing the data in the database.

The system performance indicator is collected by the controller in a certain period of time, and includes at least a data volume, a query average access data amount, a processed row number, a read row number ratio, a query average access column ratio, and a query statement. The proportion.

Further, before determining the second storage format required for storing the data in the database, the controller needs to determine whether the system performance indicator of the database meets the conversion condition set by the user, and if the system performance indicator does not meet the conversion condition set by the user, The controller does not do any processing, and the data in the database is still stored in the first storage format.

Exemplarily, if the conversion condition set by the user is any column access frequency (the number of accesses to the column/the number of times the table is accessed) reaches 80%, the column can be converted into column storage, and the controller is collected into the database. The access frequency of the nth column reaches 80%, and the controller determines that the nth column data is stored in the column storage format.

Correspondingly, when the controller determines the second storage format required for storing the data in the database, the controller needs to calculate a table in the database that needs to perform row and column conversion, and determine that the columns in the database that need to be aggregated and the database need to be separately stored. Column. E.g, Individually accessed columns with the highest frequency of columns are stored separately in columns.

Further, after the controller determines the second storage format required for storing the data in the database, the controller further determines, according to the user configuration information, converting the storage format of the data in the database from the first storage format to the second storage format. time.

Specifically, the controller may perform a storage format conversion according to the system performance indicator during the idle time of the load, or may prompt the user to perform the conversion and the second storage format after determining the second storage format required for storing the data in the database. Displayed to the user, the controller performs the conversion of the storage format when the user inputs the command.

5 1 02. The controller converts the storage format of the data in the database from the first storage format to the second storage format.

After the controller determines the second storage format required by the database to store the data, the controller converts the storage format of the data in the database from the first storage format to the second storage format.

Specifically, after the controller determines the conversion time of the second storage format and the storage format, the controller reorganizes the data in the database according to the second storage format in the buffer, when the amount of data in the buffer reaches the disk write threshold. The controller writes the data in the buffer to disk.

Further, the controller performs different processing on the data in the buffer according to the second storage format of the data in the buffer. If the second storage format of the data in the buffer is a single column storage, converting the storage format of the data in the buffer from the first storage format to the single column storage, and compressing and storing the data in the buffer; Alternatively, if the second storage format of the data in the buffer is row-column mixed storage or row storage, the storage format of the data in the buffer is converted from the first storage format to the row-column mixed storage or the row storage, and the storage is slow. The data in the flush area.

5 1 0 3. The controller determines whether the compression ratio of the database after the storage format conversion satisfies the first preset condition, and sorts the data in the database after the storage format conversion, and tests whether the sorting time satisfies the second preset condition.

After the controller converts the storage format of the data in the database from the first storage format to the second storage format, the controller needs to detect whether the conversion of the storage format is reasonable, whether Ability to optimize the database.

Specifically, the controller judges the size change of the table that has been converted into a column in the database, and tests the sorting time by a simple sorting. Because the size of the table directly reflects the compression ratio of the database, the higher the compression ratio, the higher the space utilization; the length of the sorting time reflects the amount of memory consumed by the C P U, and the column storage can improve the sorting efficiency. Therefore, the controller detects whether the conversion of the storage format is reasonable, and needs to determine whether the compression ratio of the database after the storage format conversion satisfies the first preset condition, and sorts the data in the database after the storage format conversion, and tests whether the sorting time is satisfied. The second preset condition.

The first preset condition is that the compression ratio of the database after the storage format conversion is less than or equal to the first preset threshold, and the second preset condition is that the sorting time is less than or equal to the second preset threshold.

5 1 04. If the compression ratio satisfies the first preset condition, and the sorting time satisfies the second preset condition, the controller performs external service.

Specifically, if the compression ratio satisfies the first preset condition, and the sorting time satisfies the second preset condition, it indicates that the current storage format conversion can improve the performance of the database system, and the controller performs external service.

Further, after performing external services, the controller needs to further observe the corresponding indicators to determine whether the performance of the database system is optimized, that is, whether the database system needs to be further split into columns, or to avoid splitting the columns without division.

Among them, the indicators collected by the controller include data throughput and query statement response time. The data throughput reflects whether the redundant data reading is reduced. If the data throughput is not obvious, it may need to be further split into columns. The query response time is a direct indicator to determine whether the column storage is valid.

5 1 0 5, if the compression ratio does not satisfy the first preset condition, and/or the sorting time does not satisfy the second preset condition, the controller re-executes according to the core index threshold to be set by the user.

Conversion of the secondary data storage format.

If the compression ratio does not satisfy the first preset condition, and/or the sorting time does not satisfy the second preset condition, it indicates that the second storage format determined by the controller cannot make the database system's Can be effectively improved, the controller needs to re-execute the next data storage format conversion according to the core index threshold to be set by the user to re-determine the second storage format of the data.

An embodiment of the present invention provides a data storage format conversion method. If a system performance indicator of a database storing data in a first storage format satisfies a conversion condition set by a user, the controller determines a second storage required to store data in the database. Formatting, converting the storage format of the data in the database from the first storage format to the second storage format, and then, the controller determines whether the compression ratio of the database after the storage format conversion satisfies the first preset condition, and converts the storage format After the data in the database is sorted, whether the test sorting time satisfies the second preset condition, if the compression ratio satisfies the first preset condition, and the sorting time satisfies the second preset condition, the external service is performed; or, if the compression ratio is If the first preset condition is not met, and/or the sorting time does not satisfy the second preset condition, the second storage format required to determine the data stored in the database is re-executed according to the core indicator threshold to be set by the user. Through the scheme, the controller continuously determines the optimal storage format of the data in the database system by monitoring the actual running data of the system, that is, the controller can enable the database system to dynamically determine the storage format of the data in the database system according to the load situation, and solve the current When changing the database storage format, the database administrator needs to manually modify the offline, the system storage space and low utilization rate, through the self-decision data storage format, reduce the throughput of the query statement, and at the same time improve the storage space and utilization of the system. Embodiment 4

The S 2 0 K controller collects the systematic nature of the database in which the data is stored in the first storage format.

The row, column, or row-column hybrid storage format of an existing database is completely dependent on the initial setting of the database initialization, that is, the underlying storage format of the database specified by the user when creating the database. When the user needs to change the database storage format, the DBA needs to manually modify it offline, and the system automatic adjustment optimization function is missing.

In order to solve the problem that the system cannot automatically adjust the storage format of the data in the optimized database, the present invention provides a data storage format conversion method, which can enable the database system to dynamically determine the underlying storage format of the database according to the load situation, and realize the system automatic Adjust the optimization function.

Specifically, in order to dynamically determine the underlying storage format of the database, the controller first collects system performance indicators of the database storing the data in the first storage format, so that the controller determines the database storage data according to the collected performance indicators. Storage format.

The system performance indicator includes at least the amount of data, the average amount of data accessed by the query, the number of rows processed, the proportion of rows read, the column ratio of the average access of the query, and the proportion of the query. specific,

The amount of data is an important indicator of whether or not to use the column. The larger the amount of data, the more the query is more suitable for using the column, and the amount of data is the size of the entire database;

The average amount of access data is the number of data rows used by the database per query. The scenario with a large amount of average access data is suitable for inventory.

The ratio of the number of processed lines to the number of read lines refers to the ratio of the number of data lines actually used per operation to the total number of read lines. When reading the data in the analysis database, some data is read from the disk, but actually The system does not perform related analysis operations. We expect that all the number of rows read can be processed by the system, so the higher the ratio, the more suitable for inventory;

Query the average access column ratio refers to the ratio of the columns accessed by the average query statement to the total number of columns, the smaller the ratio, the more suitable for column storage;

The proportion of query statements refers to the proportion of query operations in all database operations. The closer the ratio is to 100%, the more suitable it is.

S 2 02. The controller determines whether the system performance indicator satisfies the core indicator threshold set by the user.

After the controller collects the system performance indicators of the database storing the data in the first storage format, the controller analyzes the system performance indicators. Specifically, the controller determines whether the system performance indicator satisfies the core index threshold set by the user, that is, the controller performs decision analysis on the system performance index according to the core index threshold and the decision algorithm set by the user.

Optionally, if the core indicator threshold set by the user includes: querying the average access column ratio threshold Ta, the query statement occupying the threshold value Tq, and the processing row number occupying the read row number ratio threshold Τρ, the user-set decision algorithm It can be (query average access column ratio <Ta) and / or (query statement proportion > Tq) and / or (process row number to read row number ratio > Tp).

S203. If the system performance indicator meets the core indicator threshold, the controller determines whether the system performance indicator meets the conversion condition set by the user.

If the controller determines that the system performance indicator meets the core threshold, that is, the system performance indicator satisfies the decision algorithm set by the user, the controller determines whether the performance indicator satisfies the conversion condition set by the user. Only if the performance indicator meets the conversion conditions set by the user, the controller can determine the storage format required to store the data in the database.

Exemplarily, if the conversion condition set by the user is any column access frequency (the number of accesses to the column/the number of times the table is accessed) reaches 80%, the column can be converted into a column storage, when the controller is collected into the database. The access frequency of the n columns reaches 80%, and the controller can determine that the nth column of data is stored in the column storage format.

S204. If the system performance indicator satisfies the conversion condition, the controller determines a second storage format required for storing data in the database.

Specifically, if the system performance indicator of the database storing the data in the first storage format satisfies the conversion condition set by the user, the storage format of the data in the database may be converted, and the controller may determine the second required for storing the data in the database. Storage format.

Correspondingly, when the controller determines the second storage format required for storing the data in the database, the controller needs to calculate a table in the database that needs to perform row and column conversion, and determine that the columns in the database that need to be aggregated and the database need to be separately stored. Column. For example, columns that have the highest frequency of access to the column are stored separately in columns.

S205. The controller determines, according to the user configuration information, a conversion time for converting a storage format of the data in the database from the first storage format to the second storage format. After the controller determines the second storage format required for storing the data in the database, the controller further determines, according to the user configuration information, a conversion time for converting the storage format of the data in the database from the first storage format to the second storage format.

S 2 06. The controller converts the storage format of the data in the database from the first storage format to the second storage format.

Specifically, after the controller determines the conversion time of the second storage format and the storage format, the controller reorganizes the data in the database according to the second storage format in the buffer, when the amount of data in the buffer reaches the disk write threshold. The controller writes the data in the buffer to the disk, wherein if the first storage format is row storage, the controller first reads the data in the database by row, and then the database is in the red burst area. The data is reorganized according to the second storage format.

S2 07: The controller determines whether the compression ratio of the database after the storage format conversion satisfies the first preset condition, and sorts the data in the database after the storage format conversion, and tests whether the sorting time meets the second preset condition.

The controller converts the storage format of the data in the database from the first storage format to After the second storage format, the controller needs to detect whether the conversion of the storage format is reasonable and whether the database can be optimized.

Specifically, the controller judges the size change of the table that has been converted into a column in the database, and tests the sorting time by a simple sorting. Because the size of the table directly reflects the compression ratio of the database, the higher the compression ratio, the higher the space utilization; the length of the sorting time reflects the amount of CPU memory resources consumed, and the column storage can improve the sorting efficiency. Therefore, the controller detects whether the conversion of the storage format is reasonable, and needs to determine whether the compression ratio of the database after the storage format conversion satisfies the first preset condition, and sorts the data in the database after the storage format conversion, and tests whether the sorting time is satisfied. The second preset condition.

S 2 08. If the compression ratio satisfies the first preset condition, and the sorting time satisfies the second preset condition, the controller performs external service.

S 2 09. If the compression ratio does not satisfy the first preset condition, and/or the sorting time does not satisfy the second preset condition, the controller re-executes the conversion of the next-order data storage format according to the core index threshold to be set by the user. .

If the compression ratio does not satisfy the first preset condition, and/or the sorting time does not satisfy the second preset The condition indicates that the second storage format determined by the controller cannot effectively improve the data of the database system, and the controller needs to re-execute the conversion of the next data storage format according to the core index threshold to be set by the user. The second storage format.

Exemplarily, if the column access ratio is 90% when the system is initialized, the controller converts the column into a separate column storage. However, after the conversion, the system performance is not improved, and the controller feedbacks the column access ratio by 9. 0% is increased to 91%, the user resets the value of the column access ratio according to the feedback information, and the controller re-determines the second storage format according to the column access ratio newly set by the user.

S 2 1 0. If the system performance indicator does not satisfy the conversion condition, the controller keeps the format of the stored data in the database as the first storage format.

An embodiment of the present invention provides a data storage format conversion method. If a system performance indicator of a database storing data in a first storage format satisfies a conversion condition set by a user, the controller determines a second storage required to store data in the database. Formatting, converting the storage format of the data in the database from the first storage format to the second storage format, and then, the controller determines whether the compression ratio of the database after the storage format conversion satisfies the first preset condition, and converts the storage format After the data in the database is sorted, whether the test sorting time satisfies the second preset condition, if the compression ratio satisfies the first preset condition, and the sorting time satisfies the second preset condition, the external service is performed; or, if the compression ratio is If the first preset condition is not met, and/or the sorting time does not satisfy the second preset condition, the second storage required to determine the data stored in the database is performed according to the core indicator threshold to be set by the user in the feedback information. format. Through the scheme, the controller continuously determines the optimal storage format of the data in the database system by monitoring the actual running data of the system, that is, the controller can enable the database system to dynamically determine the storage format of the data in the database system according to the load situation, and solve the current When changing the database storage format, the database administrator needs to manually modify the offline, the system storage space and low utilization rate, through the self-decision data storage format, reduce the throughput of the query statement, and at the same time improve the storage space and utilization of the system.

It will be clearly understood by those skilled in the art that for the convenience and brevity of the description, only the division of the above functional modules is illustrated. In practical applications, The above functions are allocated by different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. For the specific working process of the system, the device and the unit described above, refer to the corresponding process in the foregoing method embodiment, and details are not described herein again.

In the several embodiments provided by the present application, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the device embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be another division manner, for example, multiple units or components may be used. Combined or can be integrated into another system, or some features can be ignored, or not executed.

The above is only the specific embodiment of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily think of changes or substitutions within the technical scope of the present invention. It should be covered by the scope of the present invention. Therefore, the scope of the invention should be determined by the scope of the appended claims.

Claims

claims

1. A controller, characterized in that it includes:

A decision-making unit, used to determine the second storage format required to store data in the database if the system performance index of the database that stores data in the first storage format meets the conversion conditions set by the user;

A storage format conversion unit configured to convert the storage format of data in the database from the first storage format to the second storage format determined by the decision-making unit;

A feedback unit, used to determine whether the compression ratio of the database after the storage format conversion meets the first preset condition, sort the data in the database after the storage format conversion, and test whether the sorting time meets the second preset condition; If the compression ratio satisfies the first preset condition, and the sorting time satisfies the second preset condition, external services are performed, or, if the compression ratio does not satisfy the first preset condition, and/or the sorting If the time does not meet the second preset condition, feedback information is sent to the decision-making unit, so that the decision-making unit re-determines the data required to store data in the database based on the core indicator threshold to be set by the user in the feedback information. Second storage format.

2. The controller according to claim 1, characterized in that,

The decision-making unit is also configured to determine whether the system performance index meets the core index threshold set by the user before determining the second storage format required to store data in the database; if the system performance index meets the If the core index threshold is determined, it is determined whether the system performance index meets the conversion condition.

3. The controller according to claim 2, characterized in that,

The decision-making unit is also configured to maintain the format of the data stored in the database as the first storage format if the system performance index does not meet the conversion condition.

4. The controller according to any one of claims 1 to 3, characterized in that the controller further includes a data collection unit,

The data collection unit is also used to collect the system performance indicators.

5. The controller according to any one of claims 1 to 4, characterized in that, the decision-making unit is also configured to determine the second storage format required to store data in the database according to user configuration. information to determine the storage of data in the database The storage format is converted from the first storage format to the second storage format at a conversion time.

6. The controller according to claim 5, characterized in that,

The storage format conversion unit is specifically configured to reorganize the data in the database in the buffer according to the second storage format and the conversion time determined by the decision-making unit. If the data in the buffer is When the amount of data reaches the disk write threshold, the data in the buffer is written to the disk.

7. The controller according to claim 6, characterized in that,

The storage format conversion unit is specifically configured to change the storage format of the data in the buffer from the first storage format if the second storage format of the data in the buffer is single-column storage. Convert to the single column storage, and compress and store the data in the buffer; or, if the second storage format of the data in the buffer is mixed row and column storage or row storage, then convert the The storage format of the data in the buffer is converted from the first storage format to the row-column mixed storage or the row storage, and the data in the buffer is stored.

8. The controller according to claim 6, wherein the controller further includes a reading unit;

The reading unit is configured to reorganize the data in the database in the buffer according to the second storage format and the conversion time by the storage format conversion unit, if the first storage format If it is row storage, the data in the database is read row by row.

9. The controller according to any one of claims 1 to 8, characterized in that the data collection unit is also configured to: if the compression ratio meets the first preset condition, and the sorting time If the second preset condition is met, after external services are provided, the data throughput and query statement response time of the database after the storage format conversion are collected.

10. The controller according to any one of claims 1 to 9, characterized in that the system performance indicators include at least the amount of data, the average amount of query data accessed, the ratio of the number of processed rows to the number of read rows, the query The average column ratio accessed, and the proportion of query statements.

1 1. A controller, characterized in that it includes:

A processor configured to determine the third data required for storing data in the database if the system performance index of the database that stores data in the first storage format meets the conversion conditions set by the user. Two storage formats; After the format converter converts the storage format of the data in the database from the first storage format to the second storage format, it is determined whether the compression ratio of the database after the storage format conversion meets the first predetermined value. Set conditions, sort the data in the database after the storage format conversion, and test whether the sorting time meets the second preset condition; if the compression ratio meets the first preset condition, and the sorting time meets the second preset condition If the preset conditions are met, external services will be provided, or, if the compression ratio does not meet the first preset conditions, and/or the sorting time does not meet the second preset conditions, then the core index to be set by the user will be The threshold value re-determines a second storage format required for storing data in the database;

A format converter, configured to convert the storage format of data in the database from the first storage format to the second storage format determined by the processor.

12. The controller according to claim 11, characterized in that,

The processor is also configured to determine whether the system performance index meets the core index threshold set by the user before determining the second storage format required to store data in the database; if the system performance index meets the If the core index threshold is determined, it is determined whether the system performance index meets the conversion condition.

13. The controller according to claim 12, characterized in that,

The processor is further configured to maintain the format of data stored in the database as the first storage format if the system performance index does not meet the conversion condition.

14. The controller according to any one of claims 1 1 to 1 3, characterized in that, the controller further includes a data collector;

The data collector is used to collect the system performance indicators.

15. The controller according to any one of claims 11 to 14, characterized in that, the processor is further configured to, after determining the second storage format required to store data in the database, According to the user configuration information, the conversion time for converting the storage format of the data in the database from the first storage format to the second storage format is determined.

16. The controller according to claim 15, characterized in that,

The format converter is specifically used to reorganize the data in the database in the buffer according to the second storage format and the conversion time, if the amount of data in the buffer reaches the disk write threshold , then the data in the buffer is written to disk.

17. The controller according to claim 16, characterized in that,

The format converter is specifically configured to convert the storage format of the data in the buffer from the first storage format if the second storage format of the data in the buffer is single column storage. For the single column storage, and compress and store the data in the buffer; or, if the second storage format of the data in the buffer is mixed row and column storage or row storage, then the buffer is The storage format of the data in the buffer is converted from the first storage format to the row-column mixed storage or the row storage, and the data in the buffer is stored.

18. The controller according to claim 16, characterized in that,

The processor is also configured to, before the storage format converter reorganizes the data in the database in the buffer according to the second storage format and the conversion time, if the first storage format If it is row storage, the data in the database is read row by row.

19. The controller according to any one of claims 11 to 18, characterized in that the data collector is also used to: if the compression ratio meets the first preset condition, and the If the sorting time satisfies the second preset condition, after external services are provided, the data throughput and query statement response time of the database after the storage format conversion are collected.

20. The controller according to any one of claims 11 to 19, characterized in that the system performance indicators at least include data volume, query average access data volume, and the ratio of the number of processed rows to the number of read rows. , the average column ratio accessed by the query, and the proportion of query statements.

2 1. A data storage format conversion method, characterized by including: Step a: If the system performance index of the database that stores data in the first storage format meets the conversion conditions set by the user, then determine whether the data stored in the database is stored in the first storage format. The secondary storage format required for the data;

Step b: Convert the storage format of the data in the database from the first storage format to the second storage format;

Step c: Determine whether the compression ratio of the database after the storage format conversion meets the first preset condition, sort the data in the database after the storage format conversion, and test whether the sorting time meets the second preset condition;

Step d: If the compression ratio meets the first preset condition, and the sorting time If the second preset condition is met, external services will be provided; or, if the compression ratio does not meet the first preset condition, and/or the sorting time does not meet the second preset condition, then according to Re-execute the above steps for the core indicator threshold to be set by the user.

22. The data storage format conversion method according to claim 21, characterized in that, before determining the second storage format required to store data in the database, the method further includes:

Determine whether the system performance indicator meets the core indicator threshold set by the user; if the system performance indicator meets the core indicator threshold, determine whether the system performance indicator meets the conversion condition.

23. The data storage format conversion method according to claim 22, characterized in that:

If the system performance index does not meet the conversion condition, the format of the data stored in the database is kept as the first storage format.

24. The data storage format conversion method according to any one of claims 20-23, characterized in that the method further includes:

Collect the system performance indicators.

25. The data storage format conversion method according to any one of claims 20 to 24, characterized in that, after determining the second storage format required to store data in the database, the method further includes:

According to the user configuration information, the conversion time to convert the storage format of the data in the database from the first storage format to the second storage format is determined.

26. The method of converting data storage format according to claim 25, characterized in that: converting the storage format of data in the database from the first storage format to the second storage format specifically includes: :

Reorganize the data in the database in the buffer according to the second storage format and the conversion moment;

If the amount of data in the buffer reaches the disk write threshold, the data in the buffer is written to the disk.

27. The data storage format conversion method according to claim 26, characterized by Specifically, writing the data in the buffer to the disk specifically includes: if the second storage format of the data in the buffer is single-column storage, then writing the data in the buffer to the disk. Convert the storage format from the first storage format to the single column storage, and compress and store the data in the buffer; or,

If the second storage format of the data in the buffer is mixed row and column storage or row storage, convert the storage format of the data in the buffer from the first storage format to the mixed row and row storage format. Store or the row store, and store the data in the buffer.

28. The data storage format conversion method according to claim 26, characterized in that, before reorganizing the data in the database in the buffer according to the second storage format and the conversion time, The method also includes:

If the first storage format is row storage, read the data in the database row by row.

29. The data storage format conversion method according to any one of claims 20 to 28, characterized in that if the compression ratio satisfies a first preset condition and the sorting time satisfies a second preset condition, Assuming the condition, after providing external services, the method further includes: collecting the data throughput and query statement response time of the database after the storage format conversion.

30. The data storage format conversion method according to any one of claims 20 to 29, characterized in that the system performance indicators at least include data volume, query average access data volume, number of processed rows to read rows The number ratio, the average column ratio accessed by the query, and the proportion of query statements.