CN109388644B - Data updating method and device - Google Patents
Data updating method and device Download PDFInfo
- Publication number
- CN109388644B CN109388644B CN201710675325.3A CN201710675325A CN109388644B CN 109388644 B CN109388644 B CN 109388644B CN 201710675325 A CN201710675325 A CN 201710675325A CN 109388644 B CN109388644 B CN 109388644B
- Authority
- CN
- China
- Prior art keywords
- data
- temporary table
- storage partition
- updated
- updated data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Abstract
The invention discloses a data updating method and a device, which relate to the technical field of data processing and mainly aim to reduce data updating time and shorten the inaccessible time of a read-only database, and the main technical scheme of the invention is as follows: creating a temporary table with a preset data structure, wherein the data structure of the temporary table is the same as that of a formal table in a read-only database; when receiving updated data, loading the updated data into the temporary table according to a storage partition corresponding to the updated data and the data occupation condition in the temporary table; judging whether the data in the temporary table meet a preset condition or not; and if so, replacing the data in the temporary table into a formal table of the read-only database. The invention is mainly used for updating data.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a data updating method and apparatus.
Background
With the rapid development of computer technology, data has penetrated into various industries and business function fields at present, and the arrival of the cloud era attracts more attention to big data.
The generation of the database solves the problem of data storage to a great extent, and under the scene of more queries and less updates in a distributed environment, the read-only database is more suitable for data storage, and unlike the traditional relational database, the read-only database is in column storage and does not support the update operation of data in a partition.
At present, the update operation of the read-only database usually adopts a file replacement mode, and all generated updated data entries are placed in another file, so as to cover the original updated data file. However, in a scenario where the update frequency of the read-only database is high, the above update method may overwrite the originally updated data file each time, so that the data update time is long each time, and in addition, the read-only database is inaccessible and cannot perform any data operation each time during the data file overwriting process.
Disclosure of Invention
In view of the above, the present invention provides a data updating method and apparatus, and mainly aims to reduce the data updating time and shorten the inaccessible time of the read-only database.
In order to solve the above problems, the present invention mainly provides the following technical solutions:
in one aspect, an embodiment of the present invention provides a data updating method, including:
creating a temporary table with a preset data structure, wherein the data structure of the temporary table is the same as that of a formal table in a read-only database;
when receiving updated data, loading the updated data into the temporary table according to a storage partition corresponding to the updated data and the data occupation condition in the temporary table;
judging whether the data in the temporary table meet a preset condition or not;
and if so, replacing the data in the temporary table into a formal table of the read-only database.
Further, the loading the updated data into the temporary table according to the storage partition corresponding to the updated data and the data occupation condition in the temporary table includes:
judging whether a storage partition corresponding to the updated data is the storage partition where the temporary table is located;
and if so, loading the updated data into the temporary table.
Further, the loading the updated data into the temporary table according to the storage partition corresponding to the updated data and the data occupation condition in the temporary table further includes:
if the storage partition corresponding to the updated data is not the storage partition where the temporary table is located, further judging whether the data occupation condition in the temporary table is empty;
if the data occupation condition in the temporary table is empty, adjusting the storage partition corresponding to the temporary table to be the storage partition corresponding to the updated data, and then loading the updated data into the temporary table after adjusting the storage partition, or
Deleting the temporary list, newly creating a temporary table in a storage partition corresponding to the updated data, and then loading the updated data into the newly created temporary table.
Further, the method further comprises:
if the data occupation condition in the temporary table is not null, after the original data in the temporary table is replaced to a formal table in a storage partition corresponding to the original data, clearing the data in the temporary table, adjusting the storage partition corresponding to the temporary table to be a storage partition corresponding to the updated data, and then loading the updated data into the temporary table after the adjustment of the storage partition, or
After original data in the temporary table is replaced to a formal table of a storage partition corresponding to the original data, deleting the temporary list, creating a temporary table in the storage partition corresponding to the updated data, and loading the updated data into the newly created temporary table.
Further, the judging whether the data in the temporary table meet a preset condition includes:
judging whether the accumulated updating number of the data in the temporary table is larger than a preset number or not; and/or
Judging whether the accumulated updating times of the data in the temporary table are greater than preset times or not; and/or
And judging whether the accumulated updating time of the data in the temporary table is greater than the preset time.
Further, after the replacing the data in the temporary table into the formal table of the read-only database, the method further comprises:
and clearing the data in the temporary table, and resetting the accumulated updating number, the accumulated updating times and the accumulated updating time.
In order to achieve the above object, according to another aspect of the present invention, there is provided a storage medium including a stored program, wherein when the program runs, a device on which the storage medium is located is controlled to execute the data updating method described above.
In order to achieve the above object, according to another aspect of the present invention, there is provided a processor for executing a program, wherein the program executes to perform the data updating method described above.
On the other hand, an embodiment of the present invention further provides a data updating apparatus, including:
the system comprises a creating unit, a reading unit and a processing unit, wherein the creating unit is used for creating a temporary table of a preset data structure, and the data structure of the temporary table is the same as that of a formal table in a read-only database;
the loading unit is used for loading the updated data into the temporary table according to the storage partition corresponding to the updated data and the data occupation condition in the temporary table when the updated data is received;
the judging unit is used for judging whether the data in the temporary table meet preset conditions or not;
and the replacing unit is used for replacing the data in the temporary table into a formal table of the read-only database if the data in the temporary table meets a preset condition.
Further, the loading unit includes:
the judging module is used for judging whether the storage partition corresponding to the updated data is the storage partition where the temporary table is located;
and the loading module is used for loading the updated data into the temporary table if the storage partition corresponding to the updated data is the storage partition where the temporary table is located.
Further, the loading module is specifically configured to further determine whether the data occupation status in the temporary table is empty if the storage partition corresponding to the updated data is not the storage partition where the temporary table is located;
the loading module is specifically further configured to, if the data occupancy in the temporary table is empty, adjust the storage partition corresponding to the temporary table to be the storage partition corresponding to the updated data, and then load the updated data into the temporary table after the storage partition is adjusted, or
Deleting the temporary list, newly creating a temporary table in a storage partition corresponding to the updated data, and then loading the updated data into the newly created temporary table.
Further, the loading module is specifically configured to, if the data occupancy in the temporary table is not empty, replace the original data in the temporary table with a formal table of a storage partition corresponding to the original data, empty the data in the temporary table, adjust the storage partition corresponding to the temporary table to a storage partition corresponding to the updated data, and then load the updated data into the temporary table after the adjustment of the storage partition, or
After original data in the temporary table is replaced to a formal table of a storage partition corresponding to the original data, deleting the temporary list, creating a temporary table in the storage partition corresponding to the updated data, and loading the updated data into the newly created temporary table.
Further, the determining unit is specifically configured to determine whether the accumulated updated number of data in the temporary table is greater than a preset number; and/or
The judging unit is specifically further configured to judge whether the accumulated update times of the data in the temporary table are greater than a preset time; and/or
The judging unit is specifically further configured to judge whether the accumulated update time of the data in the temporary table is greater than a preset time.
Further, the apparatus further comprises:
and the clearing unit is used for clearing the data in the temporary table and resetting the accumulated updating number, the accumulated updating times and the accumulated updating time.
By the technical scheme, the technical scheme provided by the embodiment of the invention at least has the following advantages:
according to the data updating method and device provided by the embodiment of the invention, the temporary table with the preset data structure is established, the data structure of the temporary table is the same as that of the formal table in the read-only database, when the read-only database needs data updating, the updated data is loaded into the temporary table according to the storage partition corresponding to the updated data and the data occupation condition in the temporary table, the updated data can be accurately loaded into the corresponding storage partition in the read-only database, the data volume of the temporary table is smaller compared with that of the formal table in the read-only database, so that the data updating time is shortened, and further, after the data in the temporary table meets the preset condition, the data in the temporary table is replaced into the formal table of the read-only database at one time. Compared with the data updating method adopting the all-file coverage type in the prior art, the data updating method and the data updating device can separate the data updating from the data query to a certain extent by creating the temporary table in the read-only database, when the error data appear in the temporary table, the error data are convenient to modify, the times of repeated operation on the data in the formal table are reduced, and meanwhile, when the data updating frequency is high, only the data operation is needed to be carried out on the temporary table, and the unavailable time of the formal table in the read-only database under the high-frequency updating is reduced.
The foregoing description is only an overview of the technical solutions of the present invention, and the embodiments of the present invention are described below in order to make the technical means of the present invention more clearly understood and to make the above and other objects, features, and advantages of the present invention more clearly understandable.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:
fig. 1 is a flowchart of a data updating method according to an embodiment of the present invention;
FIG. 2 is a flow chart of another data updating method according to an embodiment of the present invention;
FIG. 3 is a block diagram of a data update apparatus according to an embodiment of the present invention;
fig. 4 is a block diagram of another data updating apparatus according to an embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.
An embodiment of the present invention provides a data updating method, as shown in fig. 1, the method does not directly replace an original storage file, but replaces update data into a formal table of a read-only database at one time by creating a temporary table having the same data structure as the formal table in the read-only database, so as to reduce the number of times of repeated operations on the formal table, and for this embodiment of the present invention, the following specific steps are provided:
101. a temporary table of preset data structures is created.
The data structure of the temporary table is the same as that of the formal table in the read-only database, the data structure may be a representation of data, and the representation of data in different read-only databases is different, and usually, the carefully selected data structure may bring higher operation or storage efficiency. For example, the common data structure includes a plurality of arrays, linear tables, linked lists, and the like, and in the embodiment of the present invention, the preset data structure is not limited, and is specifically determined according to the data structure of the formal table in the current read-only database.
It can be understood that the read-only database is a database which can only read or query data, and does not support modification or deletion, and the read-only database has a plurality of data storage partitions, and is more suitable for an application scenario with more queries and less updates in a distributed environment.
For the embodiment of the invention, because the read-only database does not support the modification operation, if the data needs to be modified, the files in the read-only database need to be replaced in a covering manner.
102. And when receiving updated data, loading the updated data into the temporary table according to the storage partition corresponding to the updated data and the data occupation condition in the temporary table.
The updated data may be data content of a formal table in a read-only database that needs to be modified by a user, or may be data that is newly generated at a time interval, which is not limited in the embodiment of the present invention, and further, when the updated data is received, the updated data is loaded into a temporary table according to a storage partition corresponding to the updated data and a data occupation condition in the temporary table, where of course, the temporary table may be loaded multiple times, and after the newly generated data is loaded into the temporary table, if a data error is found in the temporary table, the data that is in error in the temporary table may also be modified, and the data operation in the temporary table is not limited in the embodiment of the present invention.
Wherein, the read-only database is corresponding to a plurality of different storage partitions, each partition has a corresponding partition rule, so as to facilitate data management of the read-only database, for example, according to a generation date corresponding to the data, or an area range corresponding to the data, when receiving updated data, because the updated data is not randomly stored in the read-only database, the storage partition corresponding to the updated data needs to be found, so as to ensure that the data is updated to a corresponding position in the read-only database, if the data of a certain date is found to have error data, the storage partition corresponding to the data needs to be updated as the data in the error data date partition, if the data in a certain area has new data, the storage partition corresponding to the data needs to be updated as the data in the area partition corresponding to the new data, of course, the data formats of formal tables in different storage partitions may also be different, the types of data stored in formal tables in different partitions are different, and the embodiment of the invention does not limit the partition of the storage partition of the read-only database and the data content of the formal tables in the storage partition.
Similarly, because the occupation of the data in the temporary table may affect the loading of the updated data, if other updated data currently exists in the temporary table, the updated data cannot be directly loaded into the temporary table until it is unknown whether the temporary table is the storage partition corresponding to the updated data, and if no other updated data currently exists in the temporary table, the updated data is loaded into the temporary table after it is known whether the temporary table is the storage partition corresponding to the updated data.
103. And judging whether the data in the temporary table meet a preset condition or not.
The preset condition may be whether the accumulated number of data in the temporary table meets the preset number, and/or whether the accumulated update time of data in the temporary table meets the preset time, and certainly, the preset condition may be met through manual setting by a user, or other conditions, which is not limited in the embodiment of the present invention.
Because the access of the formal table in the read-only database is not limited when the data in the temporary table is loaded, for an application scene in which the read-only database is updated frequently, such as once-a-day updating or once-an-hour updating, for the embodiment of the invention, the unavailable time of the formal table in the read-only database at a higher frequency can be reduced to a certain extent by setting the preset condition for the data in the temporary table, and further, when the data in the temporary table meets the preset condition, the data is updated into the formal table again, the data updated for multiple times can be completed once, and the data updating time is shortened.
104. And if so, replacing the data in the temporary table into a formal table of the read-only database.
When the data in the temporary table meets the preset condition, the data amount in the temporary table reaches a certain threshold, a storage partition of the data to be updated in the read-only database is further searched through the partition directory, the content in the temporary table is updated to a formal table of the corresponding storage partition in the read-only database, and if the data in the temporary table does not meet the preset condition, the data amount in the temporary table does not reach the certain threshold, and if the data in the temporary table has wrong data, the data in the temporary table can be modified.
It should be noted that, after the data in the temporary table is replaced into the read-only database, the data in the temporary table is cleared, so that the read-only database is updated with the data in the subsequent process.
The data updating method provided by the embodiment of the invention comprises the steps of establishing a temporary table with a preset data structure, wherein the data structure of the temporary table is the same as that of a formal table in a read-only database, loading updated data into the temporary table according to a storage partition corresponding to the updated data and the data occupation condition in the temporary table when the read-only database needs to update the data, and accurately loading the updated data into the corresponding storage partition in the read-only database. Compared with the data updating method adopting the all-file coverage type in the prior art, the data updating method and the data updating device can separate the data updating from the data query to a certain extent by creating the temporary table in the read-only database, when the error data appear in the temporary table, the error data are convenient to modify, the times of repeated operation on the data in the formal table are reduced, and meanwhile, when the data updating frequency is high, only the data operation is needed to be carried out on the temporary table, and the unavailable time of the formal table in the read-only database under the high-frequency updating is reduced.
In order to describe a data updating method proposed by the present invention in more detail, in particular, when updated data is received, a step of loading the updated data into a temporary table according to a storage partition corresponding to the updated data and a data occupation situation in the temporary table is provided, as shown in fig. 2, another data updating method is further provided in an embodiment of the present invention, and the specific steps of the method include:
201. a temporary table of preset data structures is created in the read-only database.
The read-only database is correspondingly provided with a plurality of storage partitions, each storage partition is stored with corresponding data, the partition form of different storage partitions can be a horizontal partition or a vertical partition, a user can know which partition contains the data required in specific query conveniently by establishing the storage partitions in the read-only database, for example, a million-row table is divided into 10 partitions, each partition contains one hundred thousand rows of data, the time for querying the partitions is only one tenth of the scanning time of the whole table, and certainly, the user can also query the data conveniently by establishing indexes for the table of one hundred thousand rows.
It should be noted that the temporary table with the preset data structure is the same as the formal table in the data structure, and the specific process of creating the temporary table may be to extract the content of each field in the formal table of the read-only database, and further create the temporary table with the same data structure as the formal table according to the content of each field in the formal table, for example, the formal table is a class achievement table, each field in the class achievement table includes a student name, a student gender, and a student achievement, and the correspondingly created temporary table also includes the same field in the class achievement table.
202. And when receiving updated data, judging whether a storage partition corresponding to the updated data is the storage partition where the temporary table is located.
Because the read-only database is correspondingly provided with a plurality of different storage partitions, the data in each partition is provided with a corresponding partition rule, thereby facilitating the data management of the read-only database, such as according to the generation date corresponding to the data, or the area range corresponding to the data, when receiving the updated data, it is necessary to find the memory partition corresponding to the updated data, since the memory partition currently corresponding to the temporary table may not be the memory area corresponding to the updated data, in order to avoid putting the updated data into the wrong storage partition, it is further possible to determine whether the storage partition corresponding to the updated data is the storage partition where the temporary table is located by looking up the partition directory of the read-only database, the partition directory corresponds to a data index in a read-only database, and records a data identifier of each partition, such as data name, data date, and data area, etc., which are not limited in this embodiment of the present invention.
203a, if yes, loading the updated data into the temporary table.
If the storage partition corresponding to the updated data is judged to be the storage partition where the temporary table is located, it can be stated that the data in the storage partition needs to be updated or the data in the storage partition has error data, and the storage partition corresponding to the temporary table is the storage partition corresponding to the data that needs to be updated, and further the updated data is directly loaded into the temporary table of the corresponding storage partition.
Correspondingly, in step 203b corresponding to step 203a, otherwise, judging whether the data occupation situation in the temporary table is empty.
It should be noted that, if it is determined that the storage partition corresponding to the updated data is not the storage partition where the temporary table is located, it is determined that the current storage partition is not the storage partition corresponding to the updated data, and in order to load the updated data into the temporary table of the correct storage partition, it is usually necessary to determine a data occupation condition in the temporary table before loading the updated data into the temporary table, that is, to determine whether there is data that has not been updated before in the temporary table.
The data occupation condition in the temporary table indicates whether data which is not updated before exists in the temporary table, for example, a storage partition corresponding to the updated data is a C area, a storage partition corresponding to the current temporary table is a B area, and the temporary table stores data of which the B area is updated, the occupation condition of the temporary table is not empty, and if the temporary table is emptied after the data of the B area of the storage partition is updated, the occupation condition of the temporary table is empty.
204b, if the data occupation condition in the temporary table is empty, adjusting the storage partition corresponding to the temporary table to be the storage partition corresponding to the updated data, and then loading the updated data into the temporary table after the storage partition is adjusted.
If the data occupation condition in the temporary table is null, it indicates that the data in the current temporary table has been updated to the storage partition corresponding to the temporary table, and the content in the temporary table is null, and in order to load the updated data to the correct storage partition, the embodiment of the present invention adjusts the storage partition corresponding to the temporary table to the storage partition corresponding to the updated data, and then loads the updated data to the temporary table.
It should be noted that, if the data structures of the formal tables corresponding to different storage partitions in the read-only database are different, for example, the data structures of the formal tables in the storage partition a and the storage partition C are different, if only the storage partition a corresponding to the temporary table is adjusted to the storage partition C, the temporary table cannot be updated in the storage partition C, the temporary list corresponding to the storage partition is further deleted, then the temporary table is newly created in the storage partition corresponding to the updated data, then the updated data is loaded into the newly created temporary table, and further it is ensured that the data structure of the newly created temporary table is consistent with the data structure of the formal table in the storage partition.
Correspondingly, if the data occupation situation in the temporary table corresponding to step 204b is not null, in step 204 b', after the original data in the temporary table is replaced to the formal table in the storage partition corresponding to the original data, the data in the temporary table is cleared, the storage partition corresponding to the temporary table is adjusted to be the storage partition corresponding to the updated data, and then the updated data is loaded into the temporary table after the adjustment of the storage partition.
If the data occupation condition in the temporary table is not empty, it indicates that there may be data that has not been updated before in the current temporary table, and there may also be data that has been updated but is not cleared, further, after the original data in the temporary table is replaced with a formal table in the storage partition corresponding to the original data, the data in the temporary table is cleared, after it is ensured that the occupation condition of the temporary table is empty, the same content as that in step 204b is executed to adjust the storage partition corresponding to the temporary table to the storage partition corresponding to the updated data, and then the updated data is loaded into the temporary table.
Similarly, if the data structures of the formal tables corresponding to different storage partitions in the read-only database are different, in order to ensure that the data structures of the temporary tables are consistent with those of the formal tables in the storage partitions, after the original data in the temporary tables are replaced with the formal tables of the storage partitions corresponding to the original data, the temporary tables are deleted, the temporary tables are newly created in the storage partitions corresponding to the updated data, and then the updated data are loaded into the newly created temporary tables.
205. And judging whether the data in the temporary table meet a preset condition or not.
The preset condition may be whether the accumulated number of data in the temporary table meets the preset number, and/or whether the accumulated update time of data in the temporary table meets the preset time, and certainly, the preset condition may be met through manual setting by a user, or other conditions, which is not limited in the embodiment of the present invention.
For example, for a read-only database with real-time data update, the preset condition may be that the number of accumulated pieces of data in the temporary table reaches 100, for a read-only database with error data that needs to be updated, the preset condition may be that the number of accumulated modifications of data in the temporary table reaches 10, and of course, for a read-only database with both data update and error data update, the preset condition may be that any one of the conditions is reached when the number of accumulated pieces reaches 100 or the number of accumulated modifications reaches 10.
Because the access of the formal table in the read-only database is not limited when the data in the temporary table is loaded, for an application scene in which the read-only database is updated frequently, such as once-a-day updating or once-an-hour updating, for the embodiment of the invention, the unavailable time of the formal table in the read-only database at a higher frequency can be reduced to a certain extent by setting the preset condition for the data in the temporary table, and further, when the data in the temporary table meets the preset condition, the data is updated into the formal table again, the data updated for multiple times can be completed once, and the data updating time is shortened.
206. And if so, replacing the data in the temporary table into a formal table of the read-only database.
For the embodiment of the invention, when the data in the temporary table meets the preset condition, the data amount in the temporary table reaches a certain threshold value, the content in the temporary table is further updated to the formal table of the corresponding storage partition in the read-only database, the content needing to be updated for many times can be updated to the read-only database once, and the inaccessible time of the database is reduced.
207. And clearing the data in the temporary table, and resetting the accumulated updating number, the accumulated updating times and the accumulated updating time.
After the data in the temporary table is replaced into the read-only database, the data in the temporary table is cleared, and the number of cumulative updates, and the cumulative update time are reset, so that the read-only database is updated again in the following process.
For the embodiment of the present invention, specific application scenarios may include, but are not limited to, the following implementation manners, where the current application scenario is formal table a in a read-only database, the corresponding storage partition is 001 partition, data in the formal table is updated according to a speed of z (times/second), a time consumption of each process of updating a replacement file is t (seconds), and an updated accumulated time threshold is set to be zs and an accumulated time threshold tm, where a temporary table B having the same data structure as that of the formal table a is created in the read-only database in the embodiment of the present invention, when receiving updated data, it is determined whether the storage partition corresponding to the updated data is storage partition 001 where temporary table B is located, and if the storage partition where the current temporary table B is located is storage partition 001 where data needs to be updated, the updated data is directly loaded into temporary table B, and preliminary data update is completed, if the storage partition where the current temporary table B is located is not the storage partition 001 corresponding to the updated data but is the storage partition 002, the storage partition 001 corresponding to the updated data is obtained by looking up the data identifier of the partition directory of the read-only database, further, because the updated data of the storage partition 001 is stored in the temporary table B, the data occupation condition in the temporary table B is judged not to be null, the updated data in the temporary table B is further replaced into the formal table a of the storage partition 001, after the data update of the storage partition is completed, the data in the temporary table is emptied, and the data structure of the formal table a in the storage partition 001 is consistent with that of the formal table C corresponding to the storage partition 002, the temporary table B does not need to be deleted, the storage partition 001 corresponding to the temporary table B at present can be directly adjusted to the storage partition 002, the updated data is further loaded into the temporary table B, if new updated data exists subsequently, the execution steps are repeated, whether the accumulated updating times of the updated data in the temporary table B reach the accumulated times threshold or whether the accumulated updating time of the updated data in the temporary table B reaches the accumulated time threshold is further judged, if yes, the data in the temporary table B are replaced into the formal table C in the corresponding storage partition 002, the data in the temporary table B are emptied, and the accumulated updating times and the accumulated updating time are reset.
For the embodiment of the present invention, in the case of using the accumulated update time as the preset condition, the embodiment of the present invention only needs to perform a file replacement operation on the formal table once, where the unavailable time of the formal table is t, and compared with the prior art in which the file replacement operation on the formal table in the read-only database is performed X times and the unavailable time of the formal table is zxt, the unavailable time of the read-only database in the embodiment of the present invention is 1/z of the original time, and when z is larger, the unavailable time of the read-only database is larger, so that the data update efficiency is obviously improved.
Aiming at another data updating method provided by the embodiment of the invention, the accuracy of the position of the updated data is ensured by judging the storage partition corresponding to the updated data, if the storage partition corresponding to the updated data is not the storage partition of the temporary table, the updated data of the corresponding storage partition is loaded into the temporary table according to the data occupation condition in the temporary table, because the data volume of the temporary table is smaller compared with the formal table in the read-only database, the time for updating the data is shortened, further, when the data in the temporary table meets the preset condition, the data in the temporary table is replaced into the formal table of the read-only database at one time, when the formal table has error data, the error data is conveniently modified, the times of repeated operation on the data in the formal table are reduced, and simultaneously when the data updating frequency is higher, only the temporary table needs to be subjected to data operation, and the unavailable time of the formal table in the read-only database under high-frequency updating is reduced.
In order to achieve the above object, according to another aspect of the present invention, an embodiment of the present invention further provides a storage medium, where the storage medium includes a stored program, and when the program runs, a device on which the storage medium is located is controlled to execute the above data updating method.
In order to achieve the above object, according to another aspect of the present invention, an embodiment of the present invention further provides a processor, where the processor is configured to execute a program, where the program executes to perform the data updating method described above.
Further, as an implementation of the method shown in fig. 1 and fig. 2, another embodiment of the present invention further provides a data updating apparatus. The embodiment of the apparatus corresponds to the embodiment of the method, and for convenience of reading, details in the embodiment of the apparatus are not repeated one by one, but it should be clear that the apparatus in the embodiment can correspondingly implement all the contents in the embodiment of the method. The apparatus is used for reducing data update time and simultaneously shortening read-only database inaccessible time, and specifically, as shown in fig. 3, the apparatus includes:
the creating unit 31 may be configured to create a temporary table with a preset data structure, where the data structure of the temporary table is the same as the data structure of a formal table in the read-only database;
a loading unit 32, configured to load, when receiving updated data, the updated data into the temporary table according to a storage partition corresponding to the updated data and a data occupation condition in the temporary table;
a judging unit 33, configured to judge whether the data in the temporary table meets a preset condition;
the replacing unit 34 may be configured to replace the data in the temporary table into a formal table of the read-only database if the data in the temporary table meets a preset condition.
The embodiment of the invention provides a data updating device, which is characterized in that a temporary table with a preset data structure is established, wherein the data structure of the temporary table is the same as that of a formal table in a read-only database, when the read-only database needs data updating, the updated data is loaded into the temporary table according to a storage partition corresponding to the updated data and the data occupation condition in the temporary table, the updated data can be accurately loaded into the corresponding storage partition in the read-only database, the data volume of the temporary table is smaller than that of the formal table in the read-only database, so that the data updating time is shortened, and further, after the data in the temporary table meets preset conditions, the data in the temporary table is replaced into the formal table in the read-only database at one time. Compared with the data updating method adopting the all-file coverage type in the prior art, the data updating method and the data updating device can separate the data updating from the data query to a certain extent by creating the temporary table in the read-only database, when the error data appear in the temporary table, the error data are convenient to modify, the times of repeated operation on the data in the formal table are reduced, and meanwhile, when the data updating frequency is high, only the data operation is needed to be carried out on the temporary table, and the unavailable time of the formal table in the read-only database under the high-frequency updating is reduced.
Further, as shown in fig. 4, the apparatus further includes:
an emptying unit 35, configured to empty the data in the temporary table, and reset the accumulated update number, the accumulated update times, and the accumulated update time
Further, the loading unit 32 includes:
the determining module 321 may be configured to determine whether a storage partition corresponding to the updated data is a storage partition where the temporary table is located;
the loading module 322 may be configured to load the updated data into the temporary table if the storage partition corresponding to the updated data is the storage partition where the temporary table is located.
Further, the loading module 322 may be specifically configured to further determine whether the data occupation status in the temporary table is empty if the storage partition corresponding to the updated data is not the storage partition where the temporary table is located;
the loading module 322 may be further configured to, if the data occupancy in the temporary table is empty, adjust the storage partition corresponding to the temporary table to be the storage partition corresponding to the updated data, and then load the updated data into the temporary table after the storage partition is adjusted, or
Deleting the temporary list, newly creating a temporary table in a storage partition corresponding to the updated data, and then loading the updated data into the newly created temporary table.
Further, the loading module 322 may be specifically configured to, if the data occupation condition in the temporary table is not empty, replace the original data in the temporary table with a formal table in a storage partition corresponding to the original data, empty the data in the temporary table, adjust the storage partition corresponding to the temporary table as a storage partition corresponding to the updated data, and then load the updated data into the temporary table after adjusting the storage partition, or
After original data in the temporary table is replaced to a formal table of a storage partition corresponding to the original data, deleting the temporary list, creating a temporary table in the storage partition corresponding to the updated data, and loading the updated data into the newly created temporary table.
Further, the determining unit 33 may be specifically configured to determine whether the accumulated number of updated data in the temporary table is greater than a preset number; and/or
The determining unit 33 may be further configured to determine whether the cumulative update frequency of the data in the temporary table is greater than a preset frequency; and/or
The determining unit 33 may be further configured to determine whether an accumulated updating time of the data in the temporary table is greater than a preset time.
Aiming at another data updating device provided by the embodiment of the invention, the accuracy of the position of the updated data is ensured by judging the storage partition corresponding to the updated data, if the storage partition corresponding to the updated data is not the storage partition of the temporary table, the updated data of the corresponding storage partition is loaded into the temporary table according to the data occupation condition in the temporary table, because the data volume of the temporary table is smaller compared with the formal table in the read-only database, the time for updating the data is shortened, further, when the data in the temporary table meets the preset condition, the data in the temporary table is replaced into the formal table of the read-only database at one time, when the formal table has error data, the error data is conveniently modified, the times of repeated operation on the data in the formal table are reduced, and simultaneously when the data updating frequency is higher, only the temporary table needs to be subjected to data operation, and the unavailable time of the formal table in the read-only database under high-frequency updating is reduced.
The data updating device comprises a processor and a memory, the creating unit 31, the loading unit 32, the judging unit 33, the replacing unit 34 and the like are stored in the memory as program units, and the processor executes the program units stored in the memory to realize corresponding functions.
The processor comprises a kernel, and the kernel calls the corresponding program unit from the memory. The kernel can be set to be one or more, the data updating time is reduced by adjusting the kernel parameters, and meanwhile, the inaccessible time of the read-only database is shortened.
The memory may include volatile memory in a computer readable medium, Random Access Memory (RAM) and/or nonvolatile memory such as Read Only Memory (ROM) or flash memory (flash RAM), and the memory includes at least one memory chip.
An embodiment of the present invention provides a storage medium on which a program is stored, the program implementing the data updating method when executed by a processor.
The embodiment of the invention provides a processor, which is used for running a program, wherein the data updating method is executed when the program runs.
The embodiment of the invention provides equipment, which comprises a processor, a memory and a program which is stored on the memory and can run on the processor, wherein the processor executes the program and realizes the following steps:
a data update method, comprising: creating a temporary table with a preset data structure, wherein the data structure of the temporary table is the same as that of a formal table in a read-only database; when receiving updated data, loading the updated data into the temporary table according to a storage partition corresponding to the updated data and the data occupation condition in the temporary table; judging whether the data in the temporary table meet a preset condition or not; and if so, replacing the data in the temporary table into a formal table of the read-only database.
Further, the loading the updated data into the temporary table according to the storage partition corresponding to the updated data and the data occupation condition in the temporary table includes: judging whether a storage partition corresponding to the updated data is the storage partition where the temporary table is located; and if so, loading the updated data into the temporary table.
Further, the loading the updated data into the temporary table according to the storage partition corresponding to the updated data and the data occupation condition in the temporary table further includes: if the storage partition corresponding to the updated data is not the storage partition where the temporary table is located, further judging whether the data occupation condition in the temporary table is empty; if the data occupation condition in the temporary table is empty, adjusting the storage partition corresponding to the temporary table to be the storage partition corresponding to the updated data, then loading the updated data into the temporary table after adjusting the storage partition, or deleting the temporary table, newly creating the temporary table in the storage partition corresponding to the updated data, and then loading the updated data into the newly created temporary table.
Further, the method further comprises: if the data occupation condition in the temporary table is not null, after the original data in the temporary table is replaced to a formal table in a storage partition corresponding to the original data, clearing the data in the temporary table, adjusting the storage partition corresponding to the temporary table to be a storage partition corresponding to the updated data, and then loading the updated data to the temporary table after the adjustment of the storage partition, or after the original data in the temporary table is replaced to the formal table of the storage partition corresponding to the original data, deleting the temporary table, newly creating a temporary table in the storage partition corresponding to the updated data, and then loading the updated data to the newly created temporary table.
Further, the judging whether the data in the temporary table meet a preset condition includes: judging whether the accumulated updating number of the data in the temporary table is larger than a preset number or not; and/or judging whether the accumulated updating times of the data in the temporary table are greater than the preset times or not; and/or judging whether the accumulated updating time of the data in the temporary table is greater than the preset time.
Further, after the replacing the data in the temporary table into the formal table of the read-only database, the method further comprises: and clearing the data in the temporary table, and resetting the accumulated updating number, the accumulated updating times and the accumulated updating time.
The device herein may be a server, a PC, a PAD, a mobile phone, etc.
The present application further provides a computer program product adapted to perform program code for initializing the following method steps when executed on a data processing device: creating a temporary table with a preset data structure, wherein the data structure of the temporary table is the same as that of a formal table in a read-only database; when receiving updated data, loading the updated data into the temporary table according to a storage partition corresponding to the updated data and the data occupation condition in the temporary table; judging whether the data in the temporary table meet a preset condition or not; and if so, replacing the data in the temporary table into a formal table of the read-only database.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). The memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The above are merely examples of the present application and are not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims (9)
1. A method for updating data, comprising:
creating a temporary table with a preset data structure, wherein the data structure of the temporary table is the same as that of a formal table in a read-only database;
when receiving updated data, loading the updated data into the temporary table according to a storage partition corresponding to the updated data and the data occupation condition in the temporary table;
judging whether the data in the temporary table meet a preset condition or not;
if yes, replacing the data in the temporary table into a formal table of the read-only database;
the loading the updated data into the temporary table according to the storage partition corresponding to the updated data and the data occupation condition in the temporary table includes:
if the storage partition corresponding to the updated data is not the storage partition where the temporary table is located, further judging whether the data occupation condition in the temporary table is empty;
if the data occupation condition in the temporary table is empty, adjusting the storage partition corresponding to the temporary table to be the storage partition corresponding to the updated data, and then loading the updated data into the temporary table after adjusting the storage partition, or
And deleting the temporary table, newly creating a temporary table in a storage partition corresponding to the updated data, and loading the updated data into the newly created temporary table.
2. The method according to claim 1, wherein the loading the updated data into the temporary table according to the storage partition corresponding to the updated data and the data occupancy in the temporary table further comprises:
judging whether a storage partition corresponding to the updated data is the storage partition where the temporary table is located;
and if so, loading the updated data into the temporary table.
3. The method of claim 2, further comprising:
if the data occupation condition in the temporary table is not null, after the original data in the temporary table is replaced to a formal table in a storage partition corresponding to the original data, clearing the data in the temporary table, adjusting the storage partition corresponding to the temporary table to be a storage partition corresponding to the updated data, and then loading the updated data into the temporary table after the adjustment of the storage partition, or
After original data in the temporary table is replaced to a formal table of a storage partition corresponding to the original data, deleting the temporary table, creating a temporary table in the storage partition corresponding to the updated data, and loading the updated data into the newly created temporary table.
4. The method of claim 1, wherein the determining whether the data in the temporary table satisfies a predetermined condition comprises:
judging whether the accumulated updating number of the data in the temporary table is larger than a preset number or not; and/or
Judging whether the accumulated updating times of the data in the temporary table are greater than preset times or not; and/or
And judging whether the accumulated updating time of the data in the temporary table is greater than the preset time.
5. The method of any of claims 1-4, wherein after the replacing the data in the temporary table into the formal table of the read-only database, the method further comprises:
and clearing the data in the temporary table, and resetting the accumulated updating number, the accumulated updating times and the accumulated updating time.
6. A data update apparatus, comprising:
the system comprises a creating unit, a reading unit and a processing unit, wherein the creating unit is used for creating a temporary table of a preset data structure, and the data structure of the temporary table is the same as that of a formal table in a read-only database;
the loading unit is used for loading the updated data into the temporary table according to the storage partition corresponding to the updated data and the data occupation condition in the temporary table when the updated data is received;
the judging unit is used for judging whether the data in the temporary table meet preset conditions or not;
the replacing unit is used for replacing the data in the temporary table into a formal table of the read-only database if the data in the temporary table meets a preset condition;
the loading module is specifically configured to further determine whether the data occupation status in the temporary table is empty if the storage partition corresponding to the updated data is not the storage partition where the temporary table is located;
the loading module is specifically further configured to, if the data occupancy in the temporary table is empty, adjust the storage partition corresponding to the temporary table to be the storage partition corresponding to the updated data, and then load the updated data into the temporary table after the storage partition is adjusted, or
And deleting the temporary table, newly creating a temporary table in a storage partition corresponding to the updated data, and loading the updated data into the newly created temporary table.
7. The apparatus of claim 6, wherein the loading unit comprises:
the judging module is used for judging whether the storage partition corresponding to the updated data is the storage partition where the temporary table is located;
and the loading module is used for loading the updated data into the temporary table if the storage partition corresponding to the updated data is the storage partition where the temporary table is located.
8. A storage medium, characterized in that the storage medium comprises a stored program, wherein when the program runs, a device in which the storage medium is located is controlled to execute the data updating method of any one of claims 1 to 5.
9. A processor, configured to execute a program, wherein the program executes the data updating method according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710675325.3A CN109388644B (en) | 2017-08-09 | 2017-08-09 | Data updating method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710675325.3A CN109388644B (en) | 2017-08-09 | 2017-08-09 | Data updating method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109388644A CN109388644A (en) | 2019-02-26 |
| CN109388644B true CN109388644B (en) | 2021-10-15 |
Family
ID=65414782
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710675325.3A Active CN109388644B (en) | 2017-08-09 | 2017-08-09 | Data updating method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109388644B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110389959A (en) * | 2019-07-26 | 2019-10-29 | 中移电子商务有限公司 | A kind of data managing method, device and storage medium |
| CN110908978B (en) * | 2019-11-06 | 2022-09-13 | 中盈优创资讯科技有限公司 | Database data structure verification method and device |
| CN115455012A (en) * | 2022-11-10 | 2022-12-09 | 广东广宇科技发展有限公司 | Data acquisition method, electronic equipment and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102456024A (en) * | 2010-10-26 | 2012-05-16 | 深圳市金蝶友商电子商务服务有限公司 | Method and device for asynchronously processing newly added large data volume |
| CN102486785A (en) * | 2010-12-06 | 2012-06-06 | 金蝶软件（中国）有限公司 | Data synchronization method, device and system |
| CN103605776A (en) * | 2013-11-28 | 2014-02-26 | 北京国双科技有限公司 | Method and device for processing data of information database |
| CN105260485A (en) * | 2015-11-20 | 2016-01-20 | 杭州数梦工场科技有限公司 | Method and device for loading data |
| CN106682002A (en) * | 2015-11-05 | 2017-05-17 | 中兴通讯股份有限公司 | Database synchronization method and system, source data and target data synchronization device |
| CN106897340A (en) * | 2016-07-05 | 2017-06-27 | 阿里巴巴集团控股有限公司 | A kind of data table updating method and device |
-
2017
- 2017-08-09 CN CN201710675325.3A patent/CN109388644B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102456024A (en) * | 2010-10-26 | 2012-05-16 | 深圳市金蝶友商电子商务服务有限公司 | Method and device for asynchronously processing newly added large data volume |
| CN102486785A (en) * | 2010-12-06 | 2012-06-06 | 金蝶软件（中国）有限公司 | Data synchronization method, device and system |
| CN103605776A (en) * | 2013-11-28 | 2014-02-26 | 北京国双科技有限公司 | Method and device for processing data of information database |
| CN106682002A (en) * | 2015-11-05 | 2017-05-17 | 中兴通讯股份有限公司 | Database synchronization method and system, source data and target data synchronization device |
| CN105260485A (en) * | 2015-11-20 | 2016-01-20 | 杭州数梦工场科技有限公司 | Method and device for loading data |
| CN106897340A (en) * | 2016-07-05 | 2017-06-27 | 阿里巴巴集团控股有限公司 | A kind of data table updating method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109388644A (en) | 2019-02-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108932257B (en) | Multi-dimensional data query method and device | |
| CN106528787B (en) | query method and device based on multidimensional analysis of mass data | |
| US20170139934A1 (en) | Grid-based geofence data indexing | |
| CN106407207B (en) | Real-time newly-added data updating method and device | |
| CN109388644B (en) | Data updating method and device | |
| CN112487083B (en) | Data verification method and device | |
| CN109379398B (en) | Data synchronization method and device | |
| CN106933823B (en) | Data synchronization method and device | |
| CN111813805A (en) | Data processing method and device | |
| US9235613B2 (en) | Flexible partitioning of data | |
| CN110019298B (en) | Data processing method and device | |
| US20200042538A1 (en) | Methods and apparatus to partition a database | |
| CN110737717B (en) | Database migration method and device | |
| CN105468644A (en) | Method and device for performing query in database | |
| CN109003012B (en) | Goods location recommendation link information acquisition method, goods location recommendation method, device and system | |
| CN111784468A (en) | Account association method and device and electronic equipment | |
| CN115617799A (en) | Data storage method, device, equipment and storage medium | |
| CN110716924B (en) | Method and device for deleting expired data | |
| CN109977317B (en) | Data query method and device | |
| CN110968555B (en) | Dimension data processing method and device | |
| CN108021464B (en) | Bottom-pocketing processing method and device for application response data | |
| CN108121733B (en) | Data query method and device | |
| CN104537016A (en) | Method and device for determining zones where files are located | |
| CN114564501A (en) | Database data storage and query methods, devices, equipment and medium | |
| CN109299125B (en) | Database updating method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 100083 No. 401, 4th Floor, Haitai Building, 229 North Fourth Ring Road, Haidian District, Beijing Applicant after: Beijing Guoshuang Technology Co.,Ltd. Address before: 100086 Beijing city Haidian District Shuangyushu Area No. 76 Zhichun Road cuigongfandian 8 layer A Applicant before: Beijing Guoshuang Technology Co.,Ltd. |
|
| CB02 | Change of applicant information | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |