CN111797102A - Data writing method and device and electronic equipment - Google Patents
Data writing method and device and electronic equipment Download PDFInfo
- Publication number
- CN111797102A CN111797102A CN202010641952.7A CN202010641952A CN111797102A CN 111797102 A CN111797102 A CN 111797102A CN 202010641952 A CN202010641952 A CN 202010641952A CN 111797102 A CN111797102 A CN 111797102A
- Authority
- CN
- China
- Prior art keywords
- data
- identifier
- block
- written
- index
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2291—User-Defined Types; Storage management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the application provides a data writing method, a data writing device and electronic equipment, wherein the data writing method comprises the following steps: determining first configuration information corresponding to the identifier of the data to be written according to the identifier of the data to be written and a first preset mapping, wherein the first preset mapping is used for indicating the corresponding relation between the identifier range corresponding to the identifier of the data to be written and the configuration information, the configuration information comprises the configuration identifier and the number of data tables in a block, and the number of the data tables in the block refers to the number of the data tables in a data block corresponding to the configuration identifier; determining index information corresponding to the identifier of the data to be written according to the identifier of the data to be written and the number of data tables in the first block; and writing the data to be written according to the index information and the first configuration identifier. Based on the scheme of the application, data migration is not needed when the capacity of the database is expanded, and therefore the difficulty and the cost of database maintenance are reduced.
Description
Technical Field
The embodiment of the application relates to the technical field of databases, in particular to a data writing method and device and electronic equipment.
Background
A database is a collection of data that is stored in a way that the data can persist and can be manipulated. In consideration of the storage capacity and concurrency performance of the databases, in the related art, a preset number of databases are generally set, a preset number of data tables are set for each database, and then a mapping relationship is established according to the number of databases and the number of data tables and the identifier of the data to be written, thereby determining the storage location of the data to be written.
However, in the using process of the database, as the data volume written into the database is increased, the data volume of a single data table may exceed a bottleneck point, which may cause performance degradation of the single table. Because the storage location of the data written into the database is related to the number of the original databases and the number of the original data tables, if the number of the databases and the number of the original data tables are changed, the stored data needs to be migrated for subsequent correct reading of the stored data, the data migration is time-consuming and labor-consuming, the difficulty of database maintenance is increased, and the use of the database needs to be stopped temporarily in the process, which affects business operation.
Disclosure of Invention
In view of the above, an object of the present invention is to provide a data processing method, apparatus and computer readable storage medium, which overcome all or part of the above disadvantages.
In a first aspect, an embodiment of the present application provides a data writing method, including:
determining first configuration information corresponding to an identifier of data to be written according to the identifier of the data to be written and a first preset mapping, wherein the first preset mapping is used for indicating a corresponding relation between the identifier of at least one piece of data and at least one piece of configuration information, the first configuration information comprises a first configuration identifier and the number of data tables in a first block, and the number of the data tables in the first block is used for indicating the number of the data tables in a data block corresponding to the first configuration identifier;
determining index information corresponding to the identifier of the data to be written according to the identifier of the data to be written and the number of data tables in the first block;
and writing the data to be written according to the index information and the first configuration identifier.
Optionally, in an embodiment of the present application, the index information includes a data block index, a first data table index, and a second data table index; the determining, according to the identifier of the data to be written and the number of the data tables in the first block, index information corresponding to the identifier of the data to be written includes:
determining the data block index according to the identifier of the data to be written and the number of records in a preset block, wherein the number of records in the preset block is used for indicating the number of data records which can be stored in the data block at most;
performing modular operation according to the identification of the data to be written and the number of the data tables in the first block to obtain the index of the first data table;
and determining the second data table index according to the data block index, the first data table index and the number of data tables in the first block.
Optionally, in an embodiment of the present application, the determining the second data table index according to the data block index, the first data table index, and the number of data tables in the first block includes:
and multiplying the data block index by the number of data tables in the first block, and taking the sum of the multiplication result and the first data table index as the second data table index.
Optionally, in an embodiment of the present application, the writing the data to be written according to the index information and the first configuration identifier includes:
determining a table name of a target data table according to a preset data table naming rule, the first configuration identifier, the data block index and the first data table index contained in the index information;
determining the identifier of the target database according to the first configuration identifier, the second data table index and a second preset mapping, wherein the second preset mapping is used for indicating the corresponding relation between the configuration identifier and the identifier of the second data table index and the identifier of the database;
and writing the data to be written according to the table name of the target data table and the identification of the target database.
Optionally, in an embodiment of the present application, the determining, according to the first configuration identifier, the second data table index, and the second preset mapping, an identifier of the target database includes:
and searching in the second preset mapping according to the combination of the first configuration identifier and the second data table index to obtain the identifier of the target database.
Optionally, in an embodiment of the present application, writing the data to be written according to the table name of the target data table and the identifier of the target database includes:
determining connection information of the target database according to the identifier of the target database and a third preset mapping, wherein the third preset mapping is used for indicating the corresponding relation between the identifier of the data table and the database connection information;
and writing the data to be written according to the connection information of the target database and the table name of the target data table.
Optionally, in an embodiment of the present application, the determining, according to the identifier of the data to be written and the first preset mapping, first configuration information corresponding to the identifier of the data to be written includes:
determining an identification range corresponding to the identification of the data to be written in the first preset mapping according to the identification of the data to be written;
and determining the configuration information corresponding to the identification range as the first configuration information according to the first preset mapping.
Optionally, in an embodiment of the present application, the method further includes:
when it is determined that the current recording number of the data block corresponding to the first configuration information reaches a preset threshold, determining the sum of the upper limit of the identification range corresponding to the first configuration information and the recording number in the preset block as a new upper limit of the identification range corresponding to the first configuration information, where the current recording number is used to indicate the number of the data records already stored in the data block.
Optionally, in an embodiment of the present application, the method further includes:
adding second configuration information in the first preset mapping according to the change of the connection number of the database, and adaptively setting the number of data tables in a second block in the second configuration information;
and adding at least one identifier corresponding to the second configuration information in the first preset mapping.
In a second aspect, an embodiment of the present application provides a data reading method, including:
determining third configuration information corresponding to the identifier of the data to be read according to the identifier of the data to be read and a first preset mapping, wherein the first preset mapping is used for indicating the corresponding relation between the identifier of at least one piece of data and at least one piece of configuration information, the third configuration information comprises a third configuration identifier and the number of data tables in a third block, and the number of data tables in the third block is used for indicating the number of data tables in a data block corresponding to the third configuration identifier;
determining index information corresponding to the identifier of the data to be read according to the identifier of the data to be read and the number of data tables in the third block;
and reading the data to be read according to the index information and the third configuration identifier.
In a third aspect, an embodiment of the present application provides a data writing apparatus, including:
the device comprises a configuration information determining module, a first configuration information determining module and a second configuration information determining module, wherein the configuration information determining module is used for determining first configuration information corresponding to an identifier of data to be written according to the identifier of the data to be written and a first preset mapping, the first preset mapping is used for indicating a corresponding relation between the identifier of at least one piece of data and at least one piece of configuration information, the first configuration information comprises a first configuration identifier and the number of data tables in a first block, and the number of data tables in the first block is used for indicating the number of data tables in a data block corresponding to the first configuration identifier;
the index information determining module is used for determining the index information corresponding to the identifier of the data to be written according to the identifier of the data to be written and the number of the data tables in the first block;
and the data writing module is used for writing the data to be written according to the index information and the first configuration identifier.
In a fourth aspect, an embodiment of the present application provides a data reading apparatus, including:
the device comprises a configuration information determining module, a data reading module and a data reading module, wherein the configuration information determining module is used for determining third configuration information corresponding to an identifier of data to be read according to the identifier of the data to be read and a first preset mapping, the first preset mapping is used for indicating a corresponding relation between the identifier of at least one piece of data and the configuration information, the third configuration information comprises a third configuration identifier and the number of data tables in a third block, and the number of the data tables in the third block is used for indicating the number of the data tables in the data block corresponding to the third configuration identifier;
the index information determining module is used for determining the index information corresponding to the identifier of the data to be read according to the identifier of the data to be read and the number of the data tables in the third block;
and the data reading module is used for reading the data to be read according to the index information and the third configuration identifier.
In a fifth aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, where the memory stores program instructions, and the processor is configured to call the program instructions in the memory to execute the method according to the first aspect and the second aspect.
In a sixth aspect, embodiments of the present application provide a computer-readable storage medium storing a computer program comprising program instructions configured to, when executed by a processor, cause the processor to perform the method according to any one of the first and second aspects.
According to the data writing method provided by the embodiment of the application, the first configuration information corresponding to the identifier of the data to be written is determined according to the identifier of the data to be written and the first preset mapping, the index information corresponding to the identifier of the data to be written is determined according to the identifier of the data to be written and the number of data tables in the first block, and then the data to be written is written according to the index information. In the scheme of the application, because the first preset mapping is arranged, the first configuration information corresponding to the identifier of the data to be written can be determined according to the first preset mapping, and then the index information is determined according to the number of the data tables in the first block to route the data to be written, no matter how the number of the data tables and the number of the data tables change during capacity expansion of the database, the number of the data tables in the first block corresponding to the written data does not change, so that the data to be read can be accurately read according to the identifier of the data to be read in the data reading process, namely, data migration is not needed during capacity expansion of the database, and the difficulty and the cost of database maintenance are reduced.
Drawings
Some specific embodiments of the present application will be described in detail hereinafter by way of illustration and not limitation with reference to the accompanying drawings. The same reference numbers in the drawings identify the same or similar elements or components. Those skilled in the art will appreciate that the drawings are not necessarily drawn to scale. In the drawings:
fig. 1 is a flowchart of a data writing method according to an embodiment of the present application;
fig. 2 is an exemplary structure diagram of a data block provided in an embodiment of the present application;
fig. 3 is a schematic diagram of a preset map according to a third embodiment of the present application;
fig. 4 is a flowchart of a data reading method according to a fourth embodiment of the present application;
fig. 5 is a flowchart of an exemplary data reading or writing method according to the present embodiment;
fig. 6 is a schematic structural diagram of a data writing device according to an embodiment of the present application;
fig. 7 is a schematic structural diagram of a data reading apparatus according to an embodiment of the present application;
fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
The following further describes specific implementation of the embodiments of the present invention with reference to the drawings.
Fig. 1 is a flowchart of a data writing method according to an embodiment of the present application. The execution subject of the data writing method may be a database server. As shown in fig. 1, the data writing method includes:
s101, determining first configuration information corresponding to the identifier of the data to be written according to the identifier of the data to be written and a first preset mapping.
The first preset mapping is used for indicating a corresponding relation between at least one data identifier and at least one configuration information, the first configuration information comprises a first configuration identifier and the number of data tables in a first block, and the number of data tables in the first block is used for indicating the number of data tables in a data block corresponding to the first configuration identifier.
In this embodiment, the data to be written may be data sent by a terminal, and the terminal may be a computer, a smart phone, or the like. Of course, the data to be written may also come from other servers. And each data to be written corresponds to a preset identification ID. In the development process of the service, the service system correspondingly assigns an identification ID to each data to be written. In the process of writing and reading the database, each data carries the identification ID distributed by the service system. The Identifier may be an incremental digital Identifier, such as a Globally Unique Identifier (GUID).
Since the first preset map is used to indicate a correspondence between the identifier of the at least one data and the at least one configuration information, one configuration information may be uniquely determined in the first preset map according to the identifier of the data to be written. In a specific implementation manner, the first preset mapping may include at least one identification range and at least one configuration information, the identification ranges are stored in correspondence with the configuration information, there is no overlap between the identification ranges, and each identification range corresponds to a unique configuration information. After the identifier of the data to be written is obtained, an identifier range corresponding to the identifier of the data to be written may be determined in the first preset map according to the identifier of the data to be written, and the configuration information corresponding to the identifier range may be determined as the configuration information corresponding to the identifier of the data to be written according to the first preset map. Hereinafter, for convenience of description, configuration information corresponding to an identification of data to be written is referred to as first configuration information. The identification range may be represented in the form of an upper identification range limit and a lower identification range limit, for example, when the identification of the data to be written is a GUID, the identification range may be represented by a maximum GUID and a minimum GUID corresponding to the identification range. For example, when the identification of the data to be written is a GUID, and the identification range is represented by a maximum GUID and a minimum GUID, the first configuration information may be determined according to GUID > -min _ GUID _ angle < -max _ GUID.
Each configuration information in the first predetermined map may include a configuration identification and a number of data tables within the block. Hereinafter, for convenience of description, the configuration identifier and the number of data tables within a block included in the first configuration information are referred to as a first configuration identifier and first data table data within a block, respectively. In the first preset mapping, each configuration information corresponds to a unique configuration identifier, which may be an incremental digital identifier or other identifier. Each configuration identifier corresponds to at least one data block, and the number of data records that can be stored in each data block at most may be referred to as the number of records in a preset block. In a specific implementation manner, when the current number of records of the data block reaches the preset threshold, the upper limit of the identification range corresponding to the data block may be automatically increased by the program by the number of records in the preset block to increase the number of data blocks corresponding to the configuration identification, so as to conveniently extend the number of the data tables.
Each data block corresponding to each configuration information may contain at least one data table, and the at least one data table may be located in at least one database. For example, if the data block is composed of 4 data tables and the number of the databases is 4, the 4 data tables may be respectively located in the 4 databases, or the 4 data tables may be all located in 1 database of the 4 databases, or the 4 data tables may be distributed in 2 or 3 databases of the 4 databases, according to different specific configurations. The number of data tables contained per data block may be referred to as the number of data tables within the block. Each data table may contain a plurality of data records, and the number of data records that can be stored in each data table may be set according to the upper limit of a single table in different databases, for example, for mysql databases, the number of data records that can be stored in each data table may be set to 2^22 ^ 4194304 or so.
For example, the constituent elements of an exemplary data block are shown in fig. 2, table _ k _0, table _ k _1, and table _ k _ i respectively represent data tables in the data block, BCI represents an index of each data table in the data block, and the number of data tables in each data block is BCN. row _ k _ i _0 to row _ k _ i _ j respectively represent data records in a data table, BRI represents an index of each data record, and the number of records in each data table is BRN. The index of the data block is BI, each data block writes TBR records at most, when the TBR data records are fully written in the data block, the index of the data block is automatically increased by 1, and new data to be written can be written into the next data block.
It should be noted that the first preset mapping may be preset by the database server, and may be optionally updated by the database server during the writing process of the data. For example, when the number of connections of the database changes, the concurrency requirement of the database changes, and at this time, new configuration information may be added to the first preset map according to the new concurrency requirement of the database, so as to update the first preset map. For another example, when the number of the written data reaches the upper limit of the identification range corresponding to the specific configuration information, the upper limit of the identification range corresponding to the specific configuration information may be adjusted to update the first preset mapping. Specifically, for example, in an embodiment of the present application, when it is determined that the current number of records in the data block corresponding to the first configuration information reaches the preset threshold, the sum of the upper limit of the identification range corresponding to the first configuration information and the number of records in the preset block is determined as the new upper limit of the identification range corresponding to the first configuration information, thereby implementing the update of the first preset mapping.
It should be noted that the first preset mapping may be represented by a configuration table, a file or other representations. For example, when the first preset mapping is represented in the form of a configuration table, the identification range and the configuration information corresponding to the identification range may be represented as a configuration record of the configuration table. The number of records in the configuration table increases by 1 for each additional piece of configuration information.
It should be noted that, besides the configuration identifier and the number of data tables in the block, the configuration information may also include other information, such as the number of data records that can be stored in each data table in the data block, and the data table naming rule.
S102, determining index information corresponding to the identification of the data to be written according to the identification of the data to be written and the number of data tables in the first block.
In an embodiment of the present application, the index information may include a data block index, a first data table index, and a second data table index. The data block index refers to an index of a data block corresponding to the identifier of the data to be written, the first data table index refers to an index of a data table corresponding to the identifier of the data to be written in the data block, and the second data table index refers to an index of the data table corresponding to the identifier of the data to be written in the whole database. Correspondingly, in this embodiment, determining the index information corresponding to the identifier of the data to be written according to the identifier of the data to be written and the number of data tables in the first block may include: determining a data block index according to the identification of the data to be written and the number of records in a preset block; performing modular operation on the number of the data tables in the first block according to the identifier of the data to be written to obtain a first data table index; and determining a second data table index according to the data block index, the number of the data tables in the first block and the first data table index. Determining a second data table index according to the data block index, the first data table index and the number of data tables in the first block, wherein the determining the second data table index comprises: and multiplying the data block index by the number of data tables in the first block, and taking the sum of the multiplication result and the first data table index as a second data table index.
A specific example is listed herein to describe the determination process of the data block index, the first data table index and the second data table index in detail, and it should be understood that this is only an example, and the embodiments of the present application are not limited thereto. In this example, if the identifier of the data to be written is a GUID, and the minimum value of the GUID is 1, the data block index, the first data table index, and the second data table index may be determined by a preset formula, where the preset formula is:
BI=ROUND((GUID-1)/TBR)
BCI=(GUID-1)%BCN
TCI=(BI x BCN)+BCI
wherein, TBR represents the number of records in the preset block, BCN represents the number of data tables in the first block, BI represents the index of the data block, BCI represents the index of the first data table, TCI represents the index of the second data table, ROUND represents a rounding-down function, and% represents the modular operation.
And S103, writing the data to be written according to the index information and the first configuration identifier.
Specifically, the storage location information of the data to be written may be determined according to the index information and the first configuration identifier, and then the data to be written may be written in the storage location indicated by the storage location information.
According to the data writing method provided by the embodiment of the application, the first configuration information corresponding to the identifier of the data to be written is determined according to the identifier of the data to be written and the first preset mapping, the index information corresponding to the identifier of the data to be written is determined according to the identifier of the data to be written and the number of data tables in the first block, and then the data to be written is written according to the index information. In the scheme of the application, because the first preset mapping is arranged, the first configuration information corresponding to the identifier of the data to be written can be determined according to the first preset mapping, and then the index information is determined according to the number of the data tables in the first block to route the data to be written, no matter how the number of the data tables and the number of the data tables change during capacity expansion of the database, the number of the data tables in the first block corresponding to the written data does not change, so that the data to be read can be accurately read according to the identifier of the data to be read in the data reading process, namely, data migration is not needed during capacity expansion of the database, and the difficulty and the cost of database maintenance are reduced.
On the basis of the embodiment shown in fig. 1, optionally, writing the data to be written according to the index information and the first configuration identifier may include the following steps:
s103a, determining the table name of the target data table according to the preset data table naming rule, the first configuration identifier, the data block index, the first data table index contained in the index information and the preset data table naming rule.
The target data table is a data table to which data to be written is written. The preset data table naming rule can be preset by the database server and is applicable to all data tables in the whole database. The preset data table naming rules may be stored in other storage means separately from the configuration information, and retrieved from other storage means by the database server when in use. However, in an implementation manner of the present application, the preset data table naming rule may be a part of the configuration information, and the preset data table naming rule may be obtained when the first configuration information corresponding to the identifier of the data to be written is obtained.
In a specific implementation manner of the present application, the preset data table naming rule may be represented as table _ [ CONF _ ID ] _[ BI ] _[ BCI ], where CONF _ ID represents a configuration identifier, BI represents a data block index, and BCI identifies a first data table index, and thus after determining the data block index and the first data table index, a table name of the target data table may be obtained according to the preset data table naming rule by combining the first configuration identifier. For example, the first configuration flag CONF _ ID is 1, the determined data block index BI is 2, and the determined first data table index BCI is 3, whereby the table name of the target data table can be determined to be table _1_2_ 3.
It should be noted that the preset data table naming rule can also be expressed as table _ [ BCI ] _ [ BI ]
CONF _ ID or other rule that can uniquely represent the table name of the target data table according to the data block index, the first data table index, and the first configuration identifier, which is not limited in this embodiment.
S103b, determining the identifier of the target database according to the first configuration identifier, the second data table index and a second preset mapping, wherein the second preset mapping is used for indicating the corresponding relation between the configuration identifier and the second data table index and the identifier of the database.
The target database is a database to which data to be written is written. The second preset mapping may be represented in a configuration table, file, or other form. For example, when the second preset mapping is represented in the form of a configuration table, the combination of the configuration identifier and the second data table index is used as a primary key, and each combination of the configuration identifier and the second data table index corresponds to the identifier of one database. The corresponding relationship between the configuration identifier, the second data table index and the identifier of the database may be represented as a configuration record of the configuration table, and the number of records in the configuration table is increased by 1 when one configuration identifier and the corresponding relationship between the second data table index and the identifier of the database are added. When the second preset mapping is represented in the configuration table form, after the second data table index is determined, the second preset mapping can be searched according to the combination of the second data table index and the first configuration identifier to obtain the identifier of the target database, and therefore the database server can write the data to be written into the target database according to the identifier of the target database.
S103c, writing the data to be written according to the table name of the target data table and the identification of the target database.
Specifically, after the table name of the target data table and the identifier of the target database are obtained, the data to be written may be written into the target data table in the target database according to the table name of the target data table and the identifier of the target database.
In a specific implementation manner, writing data to be written according to a table name of the target data table and an identifier of the target database may include: determining connection information of the target database according to the identifier of the target database and a third preset mapping, wherein the third preset mapping is used for indicating the corresponding relation between the identifier of the data table and the connection information of the database; and writing the data to be written according to the connection information of the target database and the table name of the target data table.
The third preset mapping may be a configuration table, a file, or other forms similar to the first preset mapping and the second preset mapping. For example, when the third preset mapping is represented in the form of a configuration table, identifiers of databases may be used as primary keys, and each identifier of a database corresponds to connection information of one database. The corresponding relation between the identifier of the data table and the connection information of the database can be represented as a configuration record of the configuration table, and the number of records in the configuration table is increased by 1 every time the corresponding relation between the identifier of the data table and the connection information of the database is increased.
The connection information of the database may refer to configuration information of the database, and may specifically include an account number, a password, a host, a port, a code, and the like of the database. After the connection information of the target database is found in the third preset mapping according to the identifier of the target database, the data to be written may be written into the target data table in the target database according to the connection information of the target database and the table name of the target data table.
In the embodiment of the application, each data table in the data block can be assigned to different databases through the second preset mapping, so that the data to be written can be dispersedly stored in different databases, and the concurrent response capability of the databases is improved. In addition, when the number of the data tables in the existing database reaches the upper limit and needs to be expanded, the newly-added data tables in the data blocks are assigned to the newly-added database, so that the number of the databases is conveniently expanded, and data migration is not needed, so that the method is suitable for scenes with high concurrency of mass data.
On the basis of the embodiment shown in fig. 1, optionally, the data writing method further includes the following steps:
and S104, adding second configuration information in the first preset mapping according to the change of the connection number of the database, and adaptively setting the number of data tables in a second block in the second configuration information.
In this embodiment, the change of the number of connections of the database causes a change of the concurrency requirement for the database, and therefore, when the change of the concurrency requirement for the database is determined according to the number of connections of the database, the second configuration information may be added to the first preset map. The second configuration information is similar to the first configuration information, for example, the second configuration information at least includes a second configuration identifier and data of data tables in the second block, wherein the number of data tables in the second block is adaptively set according to the change of the connection number of the database. Because each data table in the data block corresponding to each configuration information can be located in different databases, and the number of the data tables in the data block is different, the number of the databases in which the data to be written can be stored is different, the number of the databases is different, and the concurrent response capabilities of the databases are also different, the data is written in according to the second configuration information in the subsequent writing process, and the concurrent response capabilities of the databases can be adaptively adjusted.
Optionally, in a specific implementation manner, adding second configuration information in the first preset mapping according to the connection of the database and adaptively setting the number of data tables in the second block in the second configuration information may include:
when the number of the connections of the database is increased, adding second configuration information in the first preset mapping, and setting the number of the data tables in the second block to be larger than that of the data tables in the first block; and/or when the connection number of the database is reduced, adding second configuration information in the first preset mapping, and setting the number of the data tables in the second block to be smaller than that in the first block.
Specifically, if the number of data tables in the second block is greater than the number of data tables in the first block, when data writing is performed according to the second configuration information, it means that data to be written can be dispersedly stored in more databases, for example, the number of data tables in the first block is 4, the number of data tables in the second block is 8, when data writing is performed according to the first configuration information, data to be written can only be written into 4 databases at most, and when data writing is performed according to the second configuration information, data to be written can be written into 8 databases at most, so that the concurrent response capability of the databases is improved, and the concurrent requirement of the increased databases is met. On the contrary, if the number of the data tables in the second block is smaller than that of the data tables in the first block, when data writing is performed according to the second configuration information, the number of the data to be written which can be stored in the database is reduced, so that the concurrent response capability of the database is reduced, and the reduced concurrent requirement of the database is adapted.
And S105, adding at least one identifier corresponding to the second configuration information in the first preset mapping.
In this embodiment, the at least one identifier corresponding to the second configuration information may also be represented by an identifier range, for example, may be represented by an identifier range upper limit and an identifier range lower limit. When the identifier of the data to be written falls into the identifier range corresponding to the second configuration information, data writing can be performed according to the second configuration information. Specifically, when the identifier of the data to be written falls into the identifier range corresponding to the second configuration information, the second configuration information may be determined in the first preset map according to the identifier of the data to be written, so that the index information corresponding to the identifier of the data to be written is determined according to the identifier of the data to be written and the number of data tables in the second block. After determining the index information corresponding to the identifier of the data to be written, the data to be written may be written according to the index information in the same manner as in step S103 described above.
It should be noted that before writing the data to be written according to the index information, it is also necessary to add the corresponding relationship between the second data table index and the second configuration identifier and the identifier of the existing database in the second preset map as appropriate, or add the corresponding relationship between the second data table index and the second configuration identifier and the identifier of the new database in the second preset map, and configure the identifier of the new database to which the new database connection information points in the third preset map.
It should be noted that, in this embodiment, in order to write data using the second configuration information as soon as possible, a rule of an identifier generator for allocating an identifier to the data to be written may be adjusted, so that the identifier generated by the identifier generator may be increased to an identifier range corresponding to the second configuration information.
In the embodiment of the application, the second configuration information is added in the first mapping relation, and the number of the data tables in the second block in the second configuration information is flexibly set according to needs, so that the concurrency response capability of the database can be correspondingly adjusted, and the performance problems of database connection and reading and writing speed under a high concurrency scene can be effectively solved.
Based on the data writing method provided by the foregoing embodiment, this embodiment provides a data reading method, fig. 3 is a flowchart of the data reading method provided by the embodiment of the present application, and as described in fig. 3, the data reading method includes:
s301, determining third configuration information corresponding to the identifier of the data to be read according to the identifier of the data to be read and a first preset mapping, wherein the first preset mapping is used for indicating the corresponding relation between the identifier of at least one piece of data and at least one piece of configuration information, the third configuration information comprises the third configuration identifier and the number of data tables in a third block, and the number of data tables in the third block is used for indicating the number of data tables in a data block corresponding to the third configuration identifier;
s302, determining index information corresponding to the identifier of the data to be read according to the identifier of the data to be read and the number of data tables in the third block;
and S303, reading the data to be read according to the index information and the number of the data tables in the third block.
In the embodiment of the application, because the first preset mapping is arranged, the third configuration information corresponding to the identifier of the data to be read can be determined according to the first preset mapping, and then the index information is determined according to the number of the data tables in the third block in the third configuration information, so that the data to be read does not change no matter how the number of the databases and the number of the data tables change when the databases are expanded, the number of the data tables in the blocks in the configuration information corresponding to the written data does not change, so that the data to be read can be accurately read according to the identifier of the data to be read and the number of the data tables in the third block in the corresponding third configuration information, data migration is not needed when the databases are expanded, and the difficulty and the cost of database maintenance are reduced.
For convenience of understanding, the following describes in detail a process of writing or reading data by using the data writing method or the data reading method provided in the embodiments of the present application by using a specific example with reference to fig. 4 and 5, and it should be understood that this is merely an example, and the present embodiment is not limited thereto.
Specifically, referring to fig. 4, in this example, the first mapping relationship, the second mapping relationship, and the third mapping relationship each take the form of a configuration table, as shown by configuration table conf, configuration table tci _ db, and configuration table db _ conn in fig. 4, respectively. Specifically, in the configuration table conf, conf _ id corresponds to the configuration identifier in the first embodiment, bcn corresponds to the number of intra-block data tables in the first embodiment, brn corresponds to the number of data records that can be stored in each data table in the data block in the first embodiment, table _ rule corresponds to the preset data table naming rule in the first embodiment, and min _ guid and max _ guid correspond to the upper limit and the lower limit of the identifier range in the first embodiment. In the configuration table tci _ db, tci corresponds to the second index information in the first embodiment, and db _ id corresponds to the identification of the database in the first embodiment. In the configuration table db _ conn, connstr corresponds to the connection information of the database in the first embodiment. Under the specific configuration of the first mapping relationship, the second mapping relationship, and the third mapping relationship shown in fig. 4, the data writing method includes the following steps:
s501, obtaining the identifier GUID of the data to be read or written.
S502, searching conf _ id, bcn, brn and table _ rule in the configuration table conf according to the identification GUID.
Specifically, the conf _ id and the corresponding bcn, brn, and table _ rule that satisfy the following conditions may be searched in the configuration table conf according to the identification GUID: GUID > min _ GUID ANDGUID < > max _ GUID. the table _ rule may be related to the configuration identification CONF _ ID, the data block index BI, and the first data table index BCI, for example, table _ [ CONF _ ID ] _ [ BI ] _[ BCI ].
S503, determining a data block index BI, a first data table index BCI and a second data table index TCI according to the GUID identification, the configuration identification conf _ id and the number bcn of data tables in the block.
Specifically, the identification GUID, the configuration identification conf _ id, and the intra-block data table number bcn may be substituted into the following preset formulas to determine BI, BCI, and TCI, where the preset formulas are:
BI=ROUND((GUID-1)/TBR)
BCI=(GUID-1)%BCN
TCI=(BI x BCN)+BCI
wherein, TBR represents the number of records in the preset block, BCN represents the number of data tables in the block, ROUND represents a down-rounding function, and% represents a modular operation.
S504, replacing [ CONF _ ID ], [ BI ] and [ BCI ] in the table _ rule with the data block index BI, the first data table index BCI and the CONF _ ID, and determining the table name of the target data table.
S505, the identification db _ id of the target database is searched in the configuration table TCI _ db according to the second data table index TCI and conf _ id.
S506, according to the identification db _ id of the target database, the connection information connstr of the target database is in the configuration table db _ conn.
And S507, connecting the database according to the table name of the target data table and the connection information connstr of the target database, and reading or writing data.
In addition, if the database concurrent response capability needs to be added, a new configuration record may be added to the configuration table conf, and a larger BCN and corresponding parameters such as min _ guid and max _ guid may be set. Conversely, if the concurrent response capability of the database needs to be reduced, a new configuration record may be added to the configuration conf, and a smaller BCN and corresponding parameters such as min _ guid and max _ guid may be set. Thus, when the identification GUID satisfies the new GUID interval, the relevant operation will be automatically performed according to the new configuration rule. If it is desired to use the new rules as soon as possible, the rules of the GUID generator may be adjusted to increase to the GUID interval of the new configuration.
It should be noted that after adding a new configuration record to the configuration table conf, a corresponding relationship between a new conf _ id and a db _ id needs to be added to the configuration table tci _ db, and as the case may be, the new conf _ id may be assigned to an existing db _ id, or the new conf _ id may be assigned to a new db _ id, and a new database connection information connstr is added to the configuration table and assigned to the new db _ id.
In the embodiment of the present application, by setting the configuration table conf, the configuration table tci _ db, and the configuration table db _ conn, and configuring these three configuration tables, it is possible to implement that data migration is not performed when the database is expanded, and by adjusting configuration parameters in the three configuration tables, it is possible to adjust the concurrent response capability of the database, and the method is suitable for a scenario in which a large amount of data is highly concurrent.
Fig. 6 is a schematic structural diagram of a data writing device according to an embodiment of the present application. As shown in fig. 6, the data writing apparatus includes:
the configuration information determining module 61 is configured to determine, according to an identifier of the data to be written and a first preset map, first configuration information corresponding to the identifier of the data to be written, where the first preset map is used to indicate a correspondence between an identifier of at least one piece of data and the at least one piece of configuration information, the first configuration information includes the first configuration identifier and the number of data tables in a first block, and the number of data tables in the first block is used to indicate the number of data tables in a data block corresponding to the first configuration identifier;
the index information determining module 62 is configured to determine, according to the identifier of the data to be written and the number of the data tables in the first block, index information corresponding to the identifier of the data to be written;
and a data writing module 63, configured to write the data to be written according to the index information and the first configuration identifier.
The data writing device provided in this embodiment is used to implement the data writing method provided in the foregoing method embodiment, and has the beneficial effects of the corresponding method embodiment, which are not described herein again. In addition, the functional implementation of each module in the data writing device of this embodiment can refer to the description of the corresponding part of the foregoing method embodiment, and is not repeated herein.
Fig. 7 is a schematic structural diagram of a data reading apparatus according to an embodiment of the present application, and as shown in fig. 7, the data reading apparatus includes:
a configuration information determining module 71, configured to determine, according to an identifier of the data to be read and a first preset map, third configuration information corresponding to the identifier of the data to be read, where the first preset map is used to indicate a correspondence between the identifier of the at least one data and the at least one configuration information, the third configuration information includes the third configuration identifier and the number of data tables in a third block, and the number of data tables in the third block is used to indicate the number of data tables in a data block corresponding to the third configuration identifier;
the index information determining module 72 is configured to determine, according to the identifier of the data to be read and the number of the data tables in the third block, index information corresponding to the identifier of the data to be read;
and the data reading module 73 is configured to read data to be read according to the index information and the third configuration identifier.
The data reading apparatus provided in this embodiment is used to implement the data reading method provided in the foregoing method embodiment, and has the beneficial effects of the corresponding method embodiment, which are not described herein again. In addition, the functional implementation of each module in the data reading apparatus of this embodiment can refer to the description of the corresponding part of the foregoing embodiment, and is not repeated herein.
Fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present application, where the electronic device according to the embodiment is configured to execute a data writing method or a data reading method according to any method embodiment of the present application. As shown in fig. 8, the electronic device provided in this embodiment may include: a processor 81 and a memory 82, the memory 82 having stored therein program instructions, the processor 81 being configured to call the program instructions in the memory 82 to execute a data writing method or a data reading method provided by any of the method embodiments of the present application.
The processor 81 may include a central processing unit (CPU, single or multi-core), a Graphics Processing Unit (GPU), a microprocessor, an Application-Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a controller, a microcontroller, or a plurality of Integrated circuits for controlling program execution.
The Memory 82 may include, but is not limited to, a Read-Only Memory (ROM) or other type of static storage device that can store static information and instructions, a Random Access Memory (RAM) or other type of dynamic storage device that can store information and instructions, an Electrically Erasable Programmable Read-Only Memory (EEPROM), a compact disc Read-Only Memory (CD-ROM) or other optical disc storage, optical disc storage (including compact disc, laser disc, optical disc, digital versatile disc, blu-ray disc, etc.), a magnetic disc storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. The memory 82 may be provided separately or may be integrated with the processor 81.
In particular implementations, processor 81 may include one or more CPUs, as one embodiment. In a specific implementation, the electronic device may include a plurality of processors as an embodiment. Each of these processors may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. A processor herein may refer to one or more devices, circuits, and/or processing cores for processing data (e.g., computer program instructions).
For a specific execution process of the electronic device, reference may be made to any method embodiment of the present application, which achieves similar principles and technical effects, and details are not described herein again.
Embodiments of the present application provide a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements a data writing method or a data reading method provided in any method embodiment of the present application.
As will be appreciated by one skilled in the art, embodiments of the present disclosure may be provided as a method, apparatus (device), or computer program product. The above-described methods according to embodiments of the present application may be implemented in hardware, firmware, or as software or computer code storable in a recording medium such as a CD ROM, a RAM, a floppy disk, a hard disk, or a magneto-optical disk, or as computer code originally stored in a remote recording medium or a non-transitory machine-readable medium downloaded through a network and to be stored in a local recording medium, so that the methods described herein may be stored in such software processes on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware such as an ASIC or FPGA. It is understood that a computer, processor, microprocessor controller, or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code that, when accessed and executed by a computer, processor, or hardware, implements the data writing methods or data reading methods described herein. Further, when a general-purpose computer accesses code for implementing the data writing method or the data reading method illustrated herein, execution of the code converts the general-purpose computer into a special-purpose computer for executing the data writing method or the data reading method illustrated herein.
Those of ordinary skill in the art will appreciate that the various illustrative elements and method steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the embodiments of the present application.
The above embodiments are only used for illustrating the embodiments of the present application, and not for limiting the embodiments of the present application, and those skilled in the relevant art can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application, so that all equivalent technical solutions also belong to the scope of the embodiments of the present application, and the scope of patent protection of the embodiments of the present application should be defined by the claims.
Claims (12)
1. A method of writing data, comprising:
determining first configuration information corresponding to an identifier of data to be written according to the identifier of the data to be written and a first preset mapping, wherein the first preset mapping is used for indicating a corresponding relation between the identifier of at least one piece of data and at least one piece of configuration information, the first configuration information comprises a first configuration identifier and the number of data tables in a first block, and the number of the data tables in the first block is used for indicating the number of the data tables in a data block corresponding to the first configuration identifier;
determining index information corresponding to the identifier of the data to be written according to the identifier of the data to be written and the number of data tables in the first block;
and writing the data to be written according to the index information and the first configuration identifier.
2. The method of claim 1, wherein the index information comprises a data block index, a first data table index, and a second data table index; the determining, according to the identifier of the data to be written and the number of the data tables in the first block, index information corresponding to the identifier of the data to be written includes:
determining the data block index according to the identifier of the data to be written and the number of records in a preset block, wherein the number of records in the preset block is used for indicating the number of data records which can be stored in the data block at most;
performing modular operation according to the identification of the data to be written and the number of the data tables in the first block to obtain the index of the first data table;
and determining the second data table index according to the data block index, the first data table index and the number of data tables in the first block.
3. The method of claim 2, wherein determining the second data table index based on the data block index, the first data table index, and the number of data tables in the first block comprises:
and multiplying the data block index by the number of data tables in the first block, and taking the sum of the multiplication result and the first data table index as the second data table index.
4. The method according to claim 2, wherein the writing the data to be written according to the index information and the first configuration identifier comprises:
determining a table name of a target data table according to a preset data table naming rule, the first configuration identifier, the data block index and the first data table index contained in the index information;
determining the identifier of the target database according to the first configuration identifier, the second data table index and a second preset mapping, wherein the second preset mapping is used for indicating the corresponding relation between the configuration identifier and the identifier of the second data table index and the identifier of the database;
and writing the data to be written according to the table name of the target data table and the identification of the target database.
5. The method of claim 4, wherein determining the identity of the target database according to the first configuration identity, the second data table index, and a second predetermined mapping comprises:
and searching in the second preset mapping according to the combination of the first configuration identifier and the second data table index to obtain the identifier of the target database.
6. The method according to claim 4, wherein writing the data to be written according to the table name of the target data table and the identifier of the target database comprises:
determining connection information of the target database according to the identifier of the target database and a third preset mapping, wherein the third preset mapping is used for indicating the corresponding relation between the identifier of the data table and the database connection information;
and writing the data to be written according to the connection information of the target database and the table name of the target data table.
7. The method according to claim 1, wherein the determining, according to the identifier of the data to be written and the first preset mapping, the first configuration information corresponding to the identifier of the data to be written comprises:
determining an identification range corresponding to the identification of the data to be written in the first preset mapping according to the identification of the data to be written;
and determining the configuration information corresponding to the identification range as the first configuration information according to the first preset mapping.
8. The method of claim 1, further comprising:
when it is determined that the current recording number of the data block corresponding to the first configuration information reaches a preset threshold, determining the sum of the upper limit of the identification range corresponding to the first configuration information and the recording number in the preset block as a new upper limit of the identification range corresponding to the first configuration information, where the current recording number is used to indicate the number of the data records already stored in the data block.
9. The method of claim 1, further comprising:
adding second configuration information in the first preset mapping according to the change of the connection number of the database, and adaptively setting the number of data tables in a second block in the second configuration information;
and adding at least one identifier corresponding to the second configuration information in the first preset mapping.
10. A data writing apparatus, comprising:
the device comprises a configuration information determining module, a first configuration information determining module and a second configuration information determining module, wherein the configuration information determining module is used for determining first configuration information corresponding to an identifier of data to be written according to the identifier of the data to be written and a first preset mapping, the first preset mapping is used for indicating a corresponding relation between the identifier of at least one piece of data and at least one piece of configuration information, the first configuration information comprises a first configuration identifier and the number of data tables in a first block, and the number of data tables in the first block is used for indicating the number of data tables in a data block corresponding to the first configuration identifier;
the index information determining module is used for determining the index information corresponding to the identifier of the data to be written according to the identifier of the data to be written and the number of the data tables in the first block;
and the data writing module is used for writing the data to be written according to the index information and the first configuration identifier.
11. An electronic device, comprising: a processor and a memory, the memory having stored therein program instructions, the processor being configured to invoke the program instructions in the memory to perform the method recited in any of claims 1-9.
12. A computer-readable storage medium, characterized in that the computer storage medium stores a computer program comprising program instructions configured to, when executed by a processor, cause the processor to perform the method of any of claims 1-9.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202010641952.7A CN111797102A (en) | 2020-07-06 | 2020-07-06 | Data writing method and device and electronic equipment |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202010641952.7A CN111797102A (en) | 2020-07-06 | 2020-07-06 | Data writing method and device and electronic equipment |
Publications (1)
Publication Number | Publication Date |
---|---|
CN111797102A true CN111797102A (en) | 2020-10-20 |
Family
ID=72811410
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202010641952.7A Pending CN111797102A (en) | 2020-07-06 | 2020-07-06 | Data writing method and device and electronic equipment |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN111797102A (en) |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102779180A (en) * | 2012-06-29 | 2012-11-14 | 华为技术有限公司 | Operation processing method of data storage system and data storage system |
CN103714097A (en) * | 2012-10-09 | 2014-04-09 | 阿里巴巴集团控股有限公司 | Method and device for accessing database |
CN106294421A (en) * | 2015-05-25 | 2017-01-04 | 阿里巴巴集团控股有限公司 | A kind of data write, read method and device |
CN109669622A (en) * | 2017-10-13 | 2019-04-23 | 杭州海康威视系统技术有限公司 | A kind of file management method, document management apparatus, electronic equipment and storage medium |
CN110858194A (en) * | 2018-08-16 | 2020-03-03 | 北京京东尚科信息技术有限公司 | Method and device for expanding database |
-
2020
- 2020-07-06 CN CN202010641952.7A patent/CN111797102A/en active Pending
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102779180A (en) * | 2012-06-29 | 2012-11-14 | 华为技术有限公司 | Operation processing method of data storage system and data storage system |
CN103714097A (en) * | 2012-10-09 | 2014-04-09 | 阿里巴巴集团控股有限公司 | Method and device for accessing database |
CN106294421A (en) * | 2015-05-25 | 2017-01-04 | 阿里巴巴集团控股有限公司 | A kind of data write, read method and device |
CN109669622A (en) * | 2017-10-13 | 2019-04-23 | 杭州海康威视系统技术有限公司 | A kind of file management method, document management apparatus, electronic equipment and storage medium |
CN110858194A (en) * | 2018-08-16 | 2020-03-03 | 北京京东尚科信息技术有限公司 | Method and device for expanding database |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10481817B2 (en) | Methods and apparatus to optimize dynamic memory assignments in multi-tiered memory systems | |
WO2017050064A1 (en) | Memory management method and device for shared memory database | |
CN109240607B (en) | File reading method and device | |
EP4141641A2 (en) | Method and apparatus for writing data in append mode, device and storage medium | |
CN108875046A (en) | A kind of storage system access method, device and electronic equipment | |
CN111666184B (en) | Solid state drive SSD hard disk testing method and device and electronic equipment | |
CN111737212A (en) | Method and equipment for improving performance of distributed file system | |
CN115114232A (en) | Method, device and medium for enumerating historical version objects | |
CN104077082A (en) | Network voting data storage method and device | |
CN111399753B (en) | Method and device for writing pictures | |
CN115712394A (en) | Data reading and writing method and device, computer equipment and readable storage medium | |
CN115061813A (en) | Cluster resource management method, device, equipment and medium | |
US11429311B1 (en) | Method and system for managing requests in a distributed system | |
CN112650692A (en) | Heap memory allocation method, device and storage medium | |
CN112631648A (en) | Service configuration method and device, electronic equipment and storage medium | |
WO2024016789A1 (en) | Log data query method and apparatus, and device and medium | |
CN111797102A (en) | Data writing method and device and electronic equipment | |
CN108804625B (en) | LSM tree optimization method and device and computer equipment | |
CN117082046A (en) | Data uploading method, device, equipment and storage medium | |
CN111475424A (en) | Method, apparatus and computer program product for managing a storage system | |
CN115079958A (en) | Multi-node load balancing cold and hot data migration device, method, terminal and medium | |
CN115185458A (en) | Data writing method and device, storage medium and computer equipment | |
CN110399354B (en) | Partition exchange method and device for database | |
CN115904211A (en) | Storage system, data processing method and related equipment | |
CN112835490A (en) | Menu display method and device, client device and storage medium |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
RJ01 | Rejection of invention patent application after publication | ||
RJ01 | Rejection of invention patent application after publication |
Application publication date: 20201020 |