CN106933836B - Data storage method and system based on sub-tables - Google Patents
Data storage method and system based on sub-tables Download PDFInfo
- Publication number
- CN106933836B CN106933836B CN201511018398.2A CN201511018398A CN106933836B CN 106933836 B CN106933836 B CN 106933836B CN 201511018398 A CN201511018398 A CN 201511018398A CN 106933836 B CN106933836 B CN 106933836B
- Authority
- CN
- China
- Prior art keywords
- data
- sub
- date
- unit
- basic
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/219—Managing data history or versioning
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data storage method based on sub-tables, which comprises the steps of firstly storing data in a basic data table according to data generation date, then polling the data in the basic data table, starting a data sub-table thread when the data meeting preset conditions exist, transferring the data meeting the conditions to the corresponding data sub-tables, and deleting the corresponding data in the basic data table; in addition, the date of the earliest data which must be saved can be calculated according to the current date, the names of all data tables before the date are obtained, and deletion operation is carried out one by one, so that the storage requirement of a user can be met under the condition that the storage capacity does not need to be increased. After the data are stored according to the storage method provided by the invention, the storage capacity of a data table is reduced from ten million or even hundred million to one hundred thousand, the query and statistical speed is obviously improved, the query and statistical waiting time is reduced from dozens of seconds to the second, the requirements of users can be effectively met, and the user experience is improved.
Description
Technical Field
The invention belongs to the technical field of database management and application, and particularly relates to a data storage method and system based on sub-tables.
Background
With the continuous upgrade of computer equipment and the continuous development of networks, a large amount of data becomes a remarkable feature of local area networks. Generally, big data is managed and applied through a database. A cluster type data encryption device is a system for accurately recording various data, and fig. 1 is a system of a cluster type data encryption device in the prior art. As shown in fig. 1, the system is composed of a data server 11, a data sending device 12, a data encryption device cluster 13, and a local area network 14. Of course, the local area network can also perform a wider range of data networking through the internet. The whole system adopts a C/S mode network architecture, wherein the data server 11 is used as a server side in the C/S mode, and the data sending equipment 12 and the data encryption equipment cluster 13 are client sides in the C/S mode. The data server 11 processes the message from the client, and stores and summarizes the data, and the data encryption device cluster 13 is used for recording the data, encrypting the data, generating and returning encryption information uniquely identifying a piece of data to the data sending device 12.
One application scenario of the system is data collection for commercial retail enterprises. In this scenario, the data sending device 12 is a POS terminal, the data encryption device cluster 13 stores electronic invoice numbers, and the data encryption information is information such as tax control codes, invoice codes, and invoice numbers. All POS invoicing ends are concurrently connected to the data server 11 through the local area network 14, data (invoicing date, invoicing time, amount, commodity particulars, payment particulars and the like) are sent to the data server 11 before printing a receipt or an invoice after each transaction record is finished, the data server 11 records transaction data, and encryption related data such as total amount and the like are sent to the data encryption equipment cluster 13 to obtain tax control information. For large enterprises with larger business than the invoicing amount, the data server stores more data, and the storage capacity of the data server is limited, so that the data cannot be increased endlessly for any server, which provides a challenge for the data storage of the server.
Another application scenario of the system is that the business and super enterprises inquire previously issued tickets so as to carry out various statistics or ticket refunding operations. At present, many large business surpasses can already achieve provincial or even national networking of stores, and only one data server can be deployed, so that management is facilitated, and user cost is saved to a certain extent. However, one problem is that after the system runs for a period of time, the billing amount can reach tens of millions or even hundreds of millions, and if no optimization measure is taken, the waiting time for query operation according to the date is very considerable and is unacceptable to users, which puts new requirements on the query speed.
Disclosure of Invention
An object of an embodiment of the present invention is to provide a data storage method and a data query method based on sub-tables,
according to an aspect of the present invention, there is provided a data storage method based on sub-tables, the method comprising the steps of:
storing the data in the basic data table according to the data generation date;
polling data in the basic data table, and starting a data sub-table thread when the data meeting preset conditions exist;
and transferring the data for starting the data sub-table thread to the corresponding data sub-table, and deleting the corresponding data in the basic data table.
In the scheme, the preset condition is that the data information is complete, and the generation date of the data meets the requirement of the sub-table;
the data sub-tables are arranged according to the months.
In the above scheme, the method further comprises:
setting a first field for recording the minimum date of the sublist and a second field for recording the maximum date of the sublist in a database; after the data sorting thread carries out sorting for one time, updating the first field and the second field by adopting the minimum date and the minimum date in the date list;
the time limit storage thread acquires the date of the current list in the database, and the earliest date of the data needing to be stored is calculated according to the maximum effective value of the data age set by the database;
deleting the data sublist between the earliest date and the smallest date.
In the above solution, the storing the data in the basic data table according to the data generation date further includes: creating a basic data table by adopting a list data structure, and executing data adding operation at the tail of a queue of the basic data table;
the polling of the data in the basic data table further comprises: polling is performed from the head of the queue of the base data table.
In the above scheme, the transferring the data for starting the data sublist thread to the corresponding data sublist further includes:
arranging the data in groups according to dates, extracting all dates and storing the dates into a list data structure;
traversing a date list, and acquiring a branch table name of a month to which the current date belongs according to the current date of the data;
checking whether the database has a branch table corresponding to the branch table name;
when the corresponding sub-table does not exist, creating the sub-table according to the sub-table name;
and when the corresponding branch table exists, updating the data transfer flag bit corresponding to the month in the basic data table to be 1, inserting the data of the month into the corresponding branch table, and deleting the data of which the transfer flag bit is 1 in the basic data table.
According to another aspect of the present invention, there is also provided a data storage system based on sub-tables, the system comprising: the device comprises a data table creating unit, a primary storage unit, a polling unit, a table dividing unit and a deleting unit; wherein the content of the first and second substances,
the data table creating unit is used for creating a basic data table;
the preliminary storage unit is connected with the data table creating unit and used for storing the data in the basic data table according to the data generation date;
the polling unit is used for polling data in the basic data table, and starting a data sub-table thread when the data meeting a preset condition exists;
the sub-table unit is used for transferring the data for starting the sub-table threads of the data to the corresponding sub-table of the data;
the deleting unit is connected with the table dividing unit and the data table creating unit and is used for deleting the corresponding data of the finished table in the table dividing unit in the basic data table.
In the scheme, the preset condition is that the data information is complete, and the generation date of the data meets the requirement of the sub-table;
the sub-table unit is further used for arranging the data sub-tables according to months.
In the above solution, the system further includes: a threshold date setting/updating unit, a data age calculating unit; wherein the content of the first and second substances,
the threshold date setting/updating unit is used for setting a first field for recording the minimum date of the sub-tables and a second field for recording the maximum date of the sub-tables in the database; the data sorting thread is also used for updating the first field and the second field by adopting the minimum date and the minimum date in the date list after primary sorting;
the data age calculation unit is used for acquiring the date of the current list in the database and calculating the earliest date of data needing to be stored according to the maximum effective value of the data age set by the database;
the deleting unit is also connected with the threshold date setting/updating unit and the data age calculating unit and is used for deleting the data sub-table between the earliest date and the smallest date.
In the above scheme, the preliminary storage unit is further configured to create a basic data table by using a list data structure, and perform a data addition operation at the tail of a queue of the basic data table;
the polling unit is further configured to perform polling starting from the head of the queue of the basic data table.
In the above scheme, the sub-table unit includes: the sub-unit of grouping, withdraw the name sub-unit of the sub-table, inquire sub-unit of the sub-table, flag bit upgrade sub-unit, carry out sub-unit of the sub-table; wherein the content of the first and second substances,
the grouping subunit is used for grouping and arranging the data according to dates, extracting all dates and storing the dates into a list data structure;
the sub-table name extracting sub-unit is used for traversing a date list and acquiring the sub-table name of the month to which the current date belongs according to the current date of the data;
the sub-table inquiring subunit is used for checking whether a sub-table corresponding to the sub-table name exists in the database or not;
the data table creating unit is further used for creating the sub-table according to the sub-table name when the corresponding sub-table does not exist;
the flag bit updating subunit is configured to update the data transfer flag bit corresponding to the month in the basic data table to 1 when the corresponding sub-table exists;
the execution sub-table unit is used for inserting the data of the month into the corresponding sub-table;
the deleting unit is further configured to delete data with a transfer flag bit of 1 in the base data table.
According to the technical scheme, the data storage method based on the sub-tables comprises the steps of firstly storing data in a basic data table according to data generation date, then polling data in the basic data table, starting a data sub-table thread when the data meeting preset conditions exist, transferring the data meeting the conditions to the corresponding data sub-tables, and deleting the corresponding data in the basic data table; in addition, the date of the earliest data which must be saved can be calculated according to the current date, the names of all data tables before the date are obtained, and deletion operation is carried out one by one, so that the storage requirement of a user can be met under the condition that the storage capacity does not need to be increased. After the data are stored according to the storage method provided by the invention, the storage capacity of a data table is reduced from ten million or even hundred million to one hundred thousand, the query and statistical speed is obviously improved, the query and statistical waiting time is reduced from dozens of seconds to the second, the requirements of users can be effectively met, and the user experience is improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive labor.
FIG. 1 is a schematic diagram of a system of a cluster-based data encryption device in the prior art;
FIG. 2 is a flow chart of a data storage method based on sub-tables according to a first embodiment of the present invention;
FIG. 3 is a flowchart of a data storage method based on sub-tables according to a first embodiment of the present invention;
FIG. 4 is a diagram illustrating a process of performing an add and poll operation on data of a corresponding date according to a list data structure in the first embodiment of the present invention;
FIG. 5 is a schematic structural diagram of a data storage system based on sub-tables according to a second embodiment of the present invention.
Detailed Description
As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may also be present. Further, "connected" or "coupled" as used herein may include wirelessly connected or coupled. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items.
It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.
The embodiments of the present invention will be described in detail below to facilitate understanding of the embodiments of the present invention, and the embodiments described by referring to the drawings are exemplary only for the purpose of explaining the present invention and are not to be construed as limiting the present invention.
Fig. 2 is a schematic flow chart of a data storage method based on sub-tables according to a first embodiment of the present invention.
As shown in fig. 2, the data storage method based on the sub-table of the present embodiment includes the following steps:
in step S1, the data is stored in the basic data table by the data generation date.
Preferably, this step creates the base data table using a list data structure. The list is a linear table in the STL, and is characterized in that elements can be quickly added or deleted at any position, the date of the data to be transferred is stored by adopting a list data structure, and an insertion operation can be carried out at the tail of the queue, so that a data adding operation can be carried out at the tail of the queue of the basic data table; and a traversal is made from the head of the queue to batch transfer by date when transferring data.
The data are stored according to months, and the naming rule of the data table is data table basic name + YYYY + MM, wherein Y represents year, and M represents month, such as invinfo 201506. For invoice data, one data record contains the following main elements: serial number (globally unique, identifying the record), billing date, billing time, billing amount, etc. Most query statistics work is based on serial number queries or billing date periods.
And step S2, polling the data in the basic data table, and starting the data sorting process when the data meeting the preset conditions exist.
An independent data sub-table thread can start polling and execute specific sub-table processes after polling is completed.
Preferably, polling may begin at the head of the base data table, querying whether there is data that meets the branch condition. The transfer condition here may be preset, and in general, the preset condition is that the data information is complete, and the generation date of the data meets the requirement of the sublist. For invoice data, the condition of transfer is satisfied with the rules, such as the invoice data element is complete, the date is in accordance with the transfer requirement, and the transfer flag bit is 0. If not, continuing polling; if yes, screening qualified data, arranging the qualified data in groups according to dates or months, extracting all dates and storing the dates into a list data structure. Traversing the date list, firstly obtaining the branch table name according to the current month and year value, such as the basic table name + YYYY + MM.
And step S3, transferring the data for starting the data sub-table process to the corresponding data sub-table, and deleting the corresponding data in the basic data table.
Before the step, after the polling is completed, the method can further comprise the steps of arranging the data in groups according to dates, extracting all the dates and storing the dates into a list data structure; traversing a date list, and acquiring a branch table name of a month to which the current date belongs according to the current date of the data; checking whether the database has a branch table corresponding to the branch table name; and when the corresponding sub-table does not exist, creating the sub-table according to the sub-table name.
And when the corresponding branch table exists, updating the data transfer flag bit corresponding to the month in the basic data table to be 1, inserting the data of the month into the corresponding branch table, and deleting the data of which the transfer flag bit is 1 in the basic data table. The branch flag is used to indicate whether the data is to be branched, the default assignment of the branch flag is 0, which indicates that the data is not to be branched, and preferably, the branch flag is automatically assigned when the data is inserted into the basic data table.
The method can also comprise the following steps:
step S4, setting a first field for recording the minimum date of the branch table and a second field for recording the maximum date of the branch table in the database; after the data sorting thread carries out sorting for one time, updating the first field and the second field by adopting the minimum date and the minimum date in the date list; the time limit storage thread acquires the date of the current list in the database, and the earliest date of the data needing to be stored is calculated according to the maximum effective value of the data age set by the database; deleting the data sublist between the earliest date and the smallest date.
When the system runs for a long time, the data storage capacity is large, on one hand, the query speed is greatly influenced, on the other hand, due to the fact that the system storage capacity is limited, data cannot be added all the time without limitation, and for systems without special requirements, the data can be stored for 3 years generally, so that the method designed by the scheme is used for calculating the date of the earliest data which needs to be stored according to the current date, obtaining the names of all data tables before the date and deleting the data tables one by one.
In this step, an independent deadline storage thread may be set for performing the deadline storage operation. Two fields are used in the database to record the minimum month and the maximum month of the sublist, such as MinDate for the minimum month of the sublist and Maxdate for the maximum month of the sublist. When the data sorting thread performs sorting once, the maximum date and the minimum date in the date list are used to update the two field values.
The deadline storage thread firstly acquires the current system month and year value, then according to the maximum data valid value (for example, 3 years) set by the system, the month and year value DelDate of the earliest data to be stored can be calculated, and then the data sublist between MinDate and DelDate is deleted.
The application of the embodiment to the query and statistical process of data can be performed as follows:
for the query operation, generally, the query operation is performed according to the serial number of the global unique identifier of a certain piece of data, and for this, traversal query is performed according to the principle of proximity, that is, whether a data record corresponding to the serial number exists in the basic table is firstly searched, then according to the principle of date proximity, the range of the year and month in which the data record exists is calculated according to the time period required to be queried, so that the name of the data table in which the data record exists is deduced, and then all data are traversed in the table for query.
For the data operation in a certain time period, the method is that the range of the year and the month is firstly calculated according to the time period, the range of the year and the month is calculated according to the time period which needs to be counted, the range of the branch table can be obtained, then the statistics is carried out in the basic table and the branch table according to the required conditions, so that the traversal of all the branch tables can be avoided, and the response speed of the operation is improved.
It should be noted here that, because the query statistics operation thread and the deadline storage thread are two different threads executed concurrently, there is a possibility that the same data table is operated at the same time, so that a synchronization operation needs to be performed between the two threads to prevent that the data table is deleted on one side and the query operation of the same table is being performed on the other side. For this reason, a mutex CMutex is created, and no matter whether a statistical thread or a time limit storage thread is queried, before operating a data table, the CMutex needs to be acquired first; and releasing the CMutex immediately after the operation is finished. The codes between the CMutex are acquired and released to be the simplest as possible, namely, the occupied time of the CMutex is reduced as much as possible.
FIG. 3 is a flowchart of a data storage method based on sub-tables according to a first embodiment of the present invention.
As shown in fig. 3, the data storage method based on the sub-table of the present embodiment includes the following steps:
step S201, it is queried whether there is data that meets the transition condition. When there is data that meets the condition, step S202 is executed; when there is no data meeting the condition, step S201 is repeated to query the next piece of data.
The present embodiment first queries the data, and does not emphasize the call to the basic data table, and the data may be stored in the basic data table, but this is not a necessary condition and may be a single piece of data. However, when the basic data table exists, especially when the basic data table is constructed in a list data structure manner, the data generation date stores the data in the basic data table, and the batch transfer of the data is facilitated.
In step S202, a date range list in which data is located is acquired.
Step S203, the date is traversed, and whether a corresponding sublist exists is inquired. Here, the traversal may be started from the head of the queue in the data of the date list. When there is a corresponding sub-table, step S205 is executed; when there is no corresponding sub-table, step S204 is performed.
Fig. 3 is a schematic diagram illustrating a process of performing an add and poll operation on data of a corresponding date according to a list data structure in the embodiment. Here, polling starts from the head of the queue. And after the traversal is finished, the addition of the acquired new date data can be started from the tail of the queue, so that the traversal started from the head of the queue and the addition started from the tail of the queue can be ensured to be carried out simultaneously without mutual influence or conflict.
And transferring the data for starting the data sub-table thread to the corresponding data sub-table, and deleting the corresponding data in the basic data table.
In step S204, a current month table is created.
Step S205, the flag bit of the corresponding data record in the basic table is updated.
In step S206, the corresponding data is copied to the sublist of the corresponding month.
Step S207, delete the corresponding data in the base table. The base table here is the basic data table. Then, the next operation is performed, and the process returns to step S201.
According to the data storage method based on the sub-tables, firstly, data are stored in a basic data table according to data generation date, then, the data in the basic data table are polled, when the data meeting preset conditions exist, a data sub-table thread is started, the data meeting the conditions are transferred to the corresponding data sub-tables, and the corresponding data in the basic data table are deleted; in addition, the date of the earliest data which must be saved can be calculated according to the current date, the names of all data tables before the date are obtained, and deletion operation is carried out one by one, so that the storage requirement of a user can be met under the condition that the storage capacity does not need to be increased. After the data are stored according to the storage method provided by the invention, the storage capacity of a data table is reduced from ten million or even hundred million to one hundred thousand, the query and statistical speed is obviously improved, the query and statistical waiting time is reduced from dozens of seconds to the second, the requirements of users can be effectively met, and the user experience is improved.
Second embodiment
The present embodiment is a data storage system based on a branch table. FIG. 5 is a schematic structural diagram of a data storage system based on sub-tables according to a second embodiment of the present invention.
As shown in fig. 5, the data storage system based on the partial table according to the present embodiment includes: the device comprises a data table creating unit 1, a primary storage unit 2, a polling unit 3, a table dividing unit 4 and a deleting unit 5; wherein the content of the first and second substances,
the data table creating unit 1 is used for creating a basic data table;
the preliminary storage unit 2 is connected with the data table creation unit 1 and is used for storing data in the basic data table according to the data generation date;
the polling unit 3 is used for polling the data in the basic data table, and starting a data sub-table process when the data meeting preset conditions exist;
preferably, the preset condition is that the data information is complete, and the generation date of the data meets the requirement of the sublist.
The sub-table unit 4 is used for transferring the data for starting the data sub-table process to the corresponding data sub-table;
the deleting unit 5 is connected with the table dividing unit and the data table creating unit and is used for deleting the corresponding data of the finished table in the table dividing unit in the basic data table.
The system further comprises: a threshold date setting/updating unit 6, a data age calculating unit 7; wherein the content of the first and second substances,
the threshold date setting/updating unit 6 is used for setting a first field for recording the minimum date of the sublist and a second field for recording the maximum date of the sublist in the database; the data sorting thread is also used for updating the first field and the second field by adopting the minimum date and the minimum date in the date list after primary sorting;
the data age calculation unit 7 is configured to obtain dates of a current list in the database, and calculate an earliest date of data to be saved according to a maximum effective value of data age set in the database;
the deleting unit 5 is also connected to the threshold date setting/updating unit 6 and the data age calculating unit 7, and deletes the data sublist between the earliest date and the smallest date.
Preferably, the preliminary storage unit 2 is further configured to create a basic data table by using a list data structure, and perform a data addition operation at the tail of a queue of the basic data table; the polling unit 3 is further configured to perform polling starting from the head of the queue of the basic data table.
Preferably, the sub-table unit includes: the sub-unit of grouping, withdraw the name sub-unit of the sub-table, inquire sub-unit of the sub-table, flag bit upgrade sub-unit, carry out sub-unit of the sub-table; wherein the content of the first and second substances,
the grouping subunit is used for grouping and arranging the data according to dates, extracting all dates and storing the dates into a list data structure; the sub-table name extracting sub-unit is used for traversing a date list and acquiring the sub-table name of the month to which the current date belongs according to the current date of the data; the sub-table inquiring subunit is used for checking whether a sub-table corresponding to the sub-table name exists in the database or not; the data table creating unit 1 is further configured to create the sub table according to the sub table name when the corresponding sub table does not exist; the flag bit updating subunit is configured to update the data transfer flag bit corresponding to the month in the basic data table to 1 when the corresponding sub-table exists; the execution sub-table unit is used for inserting the data of the month into the corresponding sub-table; the deleting unit 3 is also configured to delete data with a transfer flag bit of 1 in the basic data table.
The data storage system based on the sub-table of the embodiment has the following beneficial effects:
1. the query statistical speed is obviously improved: the data is stored according to months, the storage capacity of one data table is reduced from ten million or even hundred million to one hundred thousand, the query speed is obviously improved, and through tests, under the condition that the data is not ideal before being sorted, the query statistical waiting time can reach dozens of seconds and seriously exceeds the limit that a user can wait; under the condition that the time is not ideal after the table division, the query and statistics waiting time can reach the second level, and the requirements of users can be effectively met.
2. The utilization rate of the storage space is improved: under the condition of no time limit storage, the storage capacity of a general data server cannot be added without limit, and the capacity of a hard disk cannot be expanded without limit. Therefore, after the storage is carried out for the term, the application requirements of the user can be met without increasing the storage capacity.
From the above description of the embodiments, it is clear to those skilled in the art that the present invention can be implemented by software plus necessary general hardware platform. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which may be stored in a storage medium, such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method according to the embodiments or some parts of the embodiments.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for apparatus or system embodiments, since they are substantially similar to method embodiments, they are described in relative terms, as long as they are described in partial descriptions of method embodiments. The above-described embodiments of the apparatus and system are merely illustrative, and the units described as separate parts may or may not be physically separate, and the parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
The above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention are included in the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (6)
1. A data storage method based on sub-tables is characterized by comprising the following steps:
storing the data in the basic data table according to the data generation date;
polling data in the basic data table, and starting a data sub-table thread when the data meeting preset conditions exist;
transferring the data for starting the data sub-table thread to a corresponding data sub-table, and deleting corresponding data in the basic data table;
the preset condition is that the data information is complete, and the generation date of the data meets the sub-table requirement;
the data sub-tables are arranged according to months;
setting a first field for recording the minimum date of the sublist and a second field for recording the maximum date of the sublist in a database; after the data sorting thread carries out sorting for one time, updating the first field and the second field by adopting the minimum date and the maximum date in the date list;
the time limit storage thread acquires the date of the current list in the database, and the earliest date of the data needing to be stored is calculated according to the maximum effective value of the data age set by the database;
deleting the data sublist between the earliest date and the smallest date.
2. The data storage method of claim 1, wherein storing data in the elementary data table by data generation date further comprises: creating a basic data table by adopting a list data structure, and executing data adding operation at the tail of a queue of the basic data table;
the polling of the data in the basic data table further comprises: polling is performed from the head of the queue of the base data table.
3. The data storage method of claim 2, wherein transferring the data for starting the data sub-table thread to the corresponding data sub-table further comprises:
arranging the data in groups according to dates, extracting all the dates and storing the dates into a list data structure;
traversing a date list, and acquiring a branch table name of a month to which the current date belongs according to the current date of the data;
checking whether the database has a branch table corresponding to the branch table name;
when the corresponding sub-table does not exist, creating the sub-table according to the sub-table name;
and when the corresponding branch table exists, updating the data transfer flag bit corresponding to the month in the basic data table to be 1, inserting the data of the month into the corresponding branch table, and deleting the data of which the transfer flag bit is 1 in the basic data table.
4. A data storage system based on sub-tables, the system comprising: the data table management system comprises a data table creating unit, a primary storage unit, a polling unit, a sub-table unit, a deleting unit, a threshold date setting/updating unit and a data age calculating unit; wherein the content of the first and second substances,
the data table creating unit is used for creating a basic data table;
the preliminary storage unit is connected with the data table creating unit and used for storing the data in the basic data table according to the data generation date;
the polling unit is used for polling data in the basic data table, and starting a data sub-table thread when the data meeting a preset condition exists;
the sub-table unit is used for transferring the data for starting the sub-table threads of the data to the corresponding sub-table of the data;
the deleting unit is connected with the sub-table unit and the data table creating unit and is used for deleting the corresponding data of the sub-table completed in the sub-table unit in the basic data table;
the preset condition is that the data information is complete, and the generation date of the data meets the sub-table requirement;
the sub-table unit is further used for arranging the data sub-tables according to months;
the threshold date setting/updating unit is used for setting a first field for recording the minimum date of the sub-tables and a second field for recording the maximum date of the sub-tables in the database; the data sorting thread is also used for updating the first field and the second field by adopting the minimum date and the maximum date in the date list after primary sorting;
the data age calculation unit is used for acquiring the date of the current list in the database and calculating the earliest date of data needing to be stored according to the maximum effective value of the data age set by the database;
the deleting unit is also connected with the threshold date setting/updating unit and the data age calculating unit and is used for deleting the data sub-table between the earliest date and the smallest date.
5. The data storage system of claim 4,
the preliminary storage unit is further used for creating a basic data table by adopting a list data structure, and executing data adding operation at the tail of a queue of the basic data table;
the polling unit is further configured to perform polling starting from the head of the queue of the basic data table.
6. The data storage system of claim 5, wherein the sub-table unit comprises: the sub-unit of grouping, withdraw the name sub-unit of the sub-table, inquire sub-unit of the sub-table, flag bit upgrade sub-unit, carry out sub-unit of the sub-table; wherein the content of the first and second substances,
the grouping subunit is used for grouping and arranging the data according to dates, extracting all dates and storing the dates into a list data structure;
the sub-table name extracting sub-unit is used for traversing a date list and acquiring the sub-table name of the month to which the current date belongs according to the current date of the data;
the sub-table inquiring subunit is used for checking whether a sub-table corresponding to the sub-table name exists in the database or not;
the data table creating unit is further used for creating the sub-table according to the sub-table name when the corresponding sub-table does not exist;
the flag bit updating subunit is configured to update the data transfer flag bit corresponding to the month in the basic data table to 1 when the corresponding sub-table exists;
the execution sub-table unit is used for inserting the data of the month into the corresponding sub-table;
the deleting unit is further configured to delete data with a transfer flag bit of 1 in the base data table.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201511018398.2A CN106933836B (en) | 2015-12-29 | 2015-12-29 | Data storage method and system based on sub-tables |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201511018398.2A CN106933836B (en) | 2015-12-29 | 2015-12-29 | Data storage method and system based on sub-tables |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106933836A CN106933836A (en) | 2017-07-07 |
| CN106933836B true CN106933836B (en) | 2021-07-06 |
Family
ID=59442305
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201511018398.2A Active CN106933836B (en) | 2015-12-29 | 2015-12-29 | Data storage method and system based on sub-tables |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106933836B (en) |
Families Citing this family (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110019436B (en) * | 2017-07-14 | 2021-04-09 | 北京国双科技有限公司 | Data import \ export method and device and data table processing method and device |
| CN107657014A (en) * | 2017-09-26 | 2018-02-02 | 上海斐讯数据通信技术有限公司 | A kind of dynamic of Large Volume Data table divides table method and system |
| CN109285080B (en) * | 2018-09-04 | 2023-04-04 | 中国平安财产保险股份有限公司 | Policy information rolling processing method and device, computer equipment and storage medium |
| CN110795431B (en) * | 2019-10-28 | 2020-12-01 | 天津同阳科技发展有限公司 | Environment monitoring data processing method, device, equipment and storage medium |
| CN112214503A (en) * | 2020-10-10 | 2021-01-12 | 深圳壹账通智能科技有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN112395270A (en) * | 2020-11-27 | 2021-02-23 | 网络通信与安全紫金山实验室 | Data management method, device, equipment and storage medium |
| CN112527828B (en) * | 2020-12-10 | 2023-03-14 | 福建新大陆支付技术有限公司 | Tax control record storage method and retrieval query method for tax control machine |
| CN115455015B (en) * | 2022-08-08 | 2024-01-26 | 中亿(深圳)信息科技有限公司 | Mass data storage method and device and storage medium |
| CN116775648A (en) * | 2023-06-05 | 2023-09-19 | 广州慧云网络科技有限公司 | Data processing method, device, equipment and medium of machine room monitoring equipment |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102073697A (en) * | 2010-12-28 | 2011-05-25 | 中兴通讯股份有限公司 | Data processing method and data processing device |
| US20150278310A1 (en) * | 2014-03-26 | 2015-10-01 | Nec Corporation | Database device |
-
2015
- 2015-12-29 CN CN201511018398.2A patent/CN106933836B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102073697A (en) * | 2010-12-28 | 2011-05-25 | 中兴通讯股份有限公司 | Data processing method and data processing device |
| US20150278310A1 (en) * | 2014-03-26 | 2015-10-01 | Nec Corporation | Database device |
Non-Patent Citations (2)
| Title |
|---|
| 基于消息中间件的数据同步更新方法;袁震;《兵工自动化》;20130715;第32卷(第7期);第93-96页 * |
| 数据库系统应用分片中间件;王亚玲 等;《计算机系统应用》;20151015;第24卷(第10期);第74-79页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106933836A (en) | 2017-07-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106933836B (en) | Data storage method and system based on sub-tables | |
| CN111046034B (en) | Method and system for managing memory data and maintaining data in memory | |
| CN108932236B (en) | File management method and device | |
| CN107977396B (en) | Method and device for updating data table of KeyValue database | |
| CN103678494B (en) | Client synchronization services the method and device of end data | |
| CN104317928A (en) | Service ETL (extraction-transformation-loading) method and service ETL system both based on distributed database | |
| CN109684369B (en) | Information updating method and device | |
| CN110134689B (en) | Target group screening method and system based on main body object label change and computer equipment | |
| CN109740129B (en) | Report generation method, device and equipment based on blockchain and readable storage medium | |
| CN110083579A (en) | Incremental data synchronous method, apparatus, computer equipment and computer storage medium | |
| CN111008521A (en) | Method and device for generating wide table and computer storage medium | |
| CN111324604A (en) | Database table processing method and device, electronic equipment and storage medium | |
| CN106649602A (en) | Way, device and server of processing business object data | |
| CN111414361A (en) | Label data storage method, device, equipment and readable storage medium | |
| CN110852559A (en) | Resource allocation method and device, storage medium and electronic device | |
| CN115544183A (en) | Data visualization method and device, computer equipment and storage medium | |
| CN109992325B (en) | Information display method, information display apparatus, storage medium, and electronic apparatus | |
| CN108182204A (en) | The processing method and processing device of data query based on house prosperity transaction multi-dimensional data | |
| CN114022188A (en) | Target crowd circling method, device, equipment and storage medium | |
| CN110362540B (en) | Data storage and visitor number acquisition method and device | |
| CN108628892B (en) | Method and device for storing ordered data, electronic equipment and readable storage medium | |
| CN107295059A (en) | The statistical system and method for service propelling amount | |
| CN115827646A (en) | Index configuration method and device and electronic equipment | |
| CN112685360B (en) | Method and device for persistence of memory data, storage medium and computer equipment | |
| CN105550347B (en) | Data processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |