CN106933836A - A kind of date storage method and system based on point table - Google Patents
A kind of date storage method and system based on point table Download PDFInfo
- Publication number
- CN106933836A CN106933836A CN201511018398.2A CN201511018398A CN106933836A CN 106933836 A CN106933836 A CN 106933836A CN 201511018398 A CN201511018398 A CN 201511018398A CN 106933836 A CN106933836 A CN 106933836A
- Authority
- CN
- China
- Prior art keywords
- data
- date
- point
- point table
- master
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/219—Managing data history or versioning
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of date storage method based on point table, in storing data in master data table by the data genaration date first, then the data in the master data table are polled, when in the presence of pre-conditioned data are met, log-on data point table thread, qualified data are transferred in corresponding data point table, and delete the corresponding data in master data table；Furthermore it is also possible to calculate the date of the earliest data that must be preserved according to current date, and the name of all tables of data before the date is obtained, and carry out deletion action one by one, so as in the case where memory capacity need not be increased, meet the amount of storage demand of user.After being stored to data according to storage method of the invention, one amount of storage of tables of data drops to 100,000 grades for even hundred million grades from millions, inquiry and Statistical Speed are obviously improved, the query statistic stand-by period dropped to second level from tens seconds, the demand of user can effectively be met, Consumer's Experience is improved.
Description
Technical field
The invention belongs to data base administration and applied technical field, and in particular to a kind of date storage method and system based on point table.
Background technology
With the continuous upgrading of computer equipment and continuing to develop for network, big data quantity turns into a notable feature of LAN.Typically, big data is managed and is applied by database.Concentrating type data encryption equipment is a kind of system for the various data of accurate recording, and Fig. 1 is a kind of concentrating type data encryption equipment system of the prior art.As shown in figure 1, the system is made up of data server 11, data transmitting equipment 12, data encryption equipment cluster 13, LAN 14.Certainly LAN here further can also carry out larger range of data network by internet.Whole system uses the C/S model network architecture, and wherein used as the server end in C/S model, data transmitting equipment 12 and data encryption equipment cluster 13 are then the client in C/S model to data server 11.Data server 11 processes the message from client, while storage, combined data, data encryption equipment cluster 13 is used for record data and to data encryption, is generated and returned to the encryption information of the data of 12 unique mark of data transmitting equipment one.
A kind of application scenarios of the system are the data acquisition of retailing enterprise.In such a scenario, data transmitting equipment 12 is POS and makes out an invoice end, the memory storage electronic invoice number of data encryption equipment cluster 13, and data encryption information is the information such as tax-control code, invoice codes, invoice number.All of POS make out an invoice end by LAN 14 it is concurrent be connected to data server 11, often complete a transaction record, printing receipt or invoice before, by data（Make out an invoice date, invoicing time, the amount of money, commodity detail, Payment Details etc.）Data server 11, the record transaction data of data server 11 are sent to, and the encryption such as Amount in Total related data is sent to data encryption equipment cluster 13 and obtain tax information.For the super this amount of making out an invoice of large-scale business compares larger enterprise, the data that data server is preserved are relatively more, because the memory capacity of data server is limited, endless increase data are impossible to any server, this data storage to server proposes a challenge.
Another application scenarios of the system are the receipt opened before the super enterprise query of business, to carry out various statistics or return ticket to operate.Many large-scale business are super at present all has been able to accomplish that shops is provincial or even national network, so can only dispose a data server, has both facilitated management, and user cost is also saved to a certain extent.But the problem brought is, when system operation for a period of time after, the amount of making out an invoice can reach millions even hundred million grades, if not taking any Optimized Measures, the stand-by period for carrying out inquiry operation according to the date will be considerable, and user is difficult to receive, and this just proposes new requirement to inquiry velocity.
The content of the invention
The purpose of the embodiment of the present invention is to provide one kind based on point table date storage method and a data query method,
According to an aspect of the invention, there is provided a kind of based on the date storage method for dividing table, methods described comprises the following steps：
In master data table being stored data in by the data genaration date；
Data in the master data table are polled, when in the presence of pre-conditioned data are met, log-on data point table thread；
The data of log-on data point table thread are transferred in corresponding data point table, and delete the corresponding data in master data table.
In such scheme, described pre-conditioned for data message is complete, the date of formation of data meets a point table requirement；
The data point table is arranged by month.
In such scheme, methods described also includes：
First field and second field on record point table maximum date on record point table minimum date are set in database；After data point table thread is once divided table, first field and the second field are updated using the minimum date in date list and minimum date；
Time limit storage thread obtains the date of the current list in database, and the data time limit maximum virtual value according to data lab setting calculates the earliest date for needing to preserve data；
Delete the data point table between the earliest date and minimum date.
It is described by the data genaration date to store data in master data table in such scheme, further for：Master data table is created using list data structures, data addition operation is performed in the tail of the queue of master data table；
The data in the master data table are polled, further for：Poll is performed since the head of the queue of master data table.
In such scheme, the data by log-on data point table thread are transferred in corresponding data point table, further for：
The data are grouped arrangement by date, and extract all dates and be stored in list data structures；
Traversal date list, the current date according to the data obtains point table name in the current date affiliated month；
Check the database with the presence or absence of point table corresponding with described point of table name；
When in the absence of the corresponding point of table, point table according to described point of table name is created；
When there is the corresponding point of table, the data transfer flag bit in master data table corresponding to the month is updated to 1, during the data in the month are inserted into corresponding point of table, while shifting the data that flag bit is 1 in deleting master data table.
According to another aspect of the present invention, a kind of data-storage system based on point table is additionally provided, it is characterised in that the system includes：Tables of data creating unit, preliminary memory cell, poll units, point table unit, deletion unit；Wherein,
The tables of data creating unit is used to create master data table；
The preliminary memory cell is connected with tables of data creating unit, in storing data in master data table by the data genaration date；
The poll units are used to be polled the data in the master data table, when in the presence of pre-conditioned data are met, log-on data point table thread；
Described point of table unit is used to be transferred to the data of log-on data point table thread in corresponding data point table；
The deletion unit is connected with described point of table unit and tables of data creating unit, and the corresponding data of the complete component list of table unit is being divided for deleting in master data table.
In such scheme, described pre-conditioned for data message is complete, the date of formation of data meets a point table requirement；
Described point of table unit is further used for being arranged the data point table by month.
In such scheme, the system also includes：Threshold value date setting/updating block, data time limit computing unit；Wherein,
The threshold value date setting/updating block is used to be set in database first field and second field on record point table maximum date on record point table minimum date；It is additionally operable to after data point table thread is once divided table, first field and the second field be updated using the minimum date in date list and minimum date；
The data time limit computing unit is used to obtain the date of the current list in database, and the data time limit maximum virtual value according to data lab setting calculates the earliest date for needing to preserve data；
The deletion unit is also connected with the threshold value date setting/updating block and the data time limit computing unit, for deleting the data point table between the earliest date and minimum date.
In such scheme, the preliminary memory cell is further used for creating master data table using list data structures, and data addition operation is performed in the tail of the queue of master data table；
The poll units are further used for performing poll since the head of the queue of master data table.
In such scheme, described point of table unit includes：Packet subelement, an extraction point table name subelement, an inquiry point table subelement, flag bit update subelement, an execution point table subelement；Wherein,
The packet subelement is used to for the data to be grouped arrangement by date, and extracts all dates and be stored in list data structures；
The extraction point table name subelement is used to travel through date list, and the current date according to the data obtains point table name in the current date affiliated month；
The inquiry point table subelement is used to check the database with the presence or absence of point table corresponding with described point of table name；
The tables of data creating unit is additionally operable to when in the absence of the corresponding point of table, point table according to described point of table name is created；
The flag bit updates subelement to be used to, when there is the corresponding point of table, the data transfer flag bit in master data table corresponding to the month is updated into 1；
In performing point table subelement and being used to for the data in the month to insert corresponding point of table；
Unit is deleted to be additionally operable to shift the data that flag bit is 1 in deletion master data table.
As can be seen from the above technical solutions, the date storage method based on point table of the embodiment of the present invention, in storing data in master data table by the data genaration date first, then the data in the master data table are polled, when in the presence of pre-conditioned data are met, log-on data point table thread, qualified data is transferred in corresponding data point table, and delete the corresponding data in master data table；Furthermore it is also possible to calculate the date of the earliest data that must be preserved according to current date, and the name of all tables of data before the date is obtained, and carry out deletion action one by one, so as in the case where memory capacity need not be increased, meet the amount of storage demand of user.After being stored to data according to storage method of the invention, one amount of storage of tables of data drops to 100,000 grades for even hundred million grades from millions, inquiry and Statistical Speed are obviously improved, the query statistic stand-by period dropped to second level from tens seconds, the demand of user can effectively be met, Consumer's Experience is lifted.
Brief description of the drawings
Technical scheme in order to illustrate more clearly the embodiments of the present invention, the accompanying drawing to be used needed for being described to embodiment below is briefly described, apparently, drawings in the following description are only some embodiments of the present invention, for those of ordinary skill in the art, without having to pay creative labor, other accompanying drawings can also be obtained according to these accompanying drawings.
Fig. 1 is concentrating type data encryption equipment system structure diagram in the prior art；
Fig. 2 is the date storage method schematic flow sheet based on point table of first embodiment of the invention；
Fig. 3 is the date storage method flow chart based on point table of first embodiment of the invention；
Fig. 4 be press in first embodiment of the invention list data structures the data on corresponding date are performed add and polling operation process schematic；
Fig. 5 is the structural representation of the data-storage system based on point table of second embodiment of the invention.
Specific embodiment
Those skilled in the art of the present technique are appreciated that unless expressly stated singulative " " used herein, " one ", " described " and " being somebody's turn to do " may also comprise plural form.It should be further understood that, used in specification of the invention wording " including " refer to the presence of the feature, integer, step, operation, element and/or component, but it is not excluded that in the presence of or add one or more other features, integer, step, operation, element, component and/or their group.It should be understood that when we claim element to be " connected " or during " coupled " to another element, it can be directly connected or coupled to other elements, or can also there is intermediary element.Additionally, " connection " used herein or " coupling " can include wireless connection or coupling.Wording "and/or" used herein includes one or more associated any cells for listing item and all combines.
Those skilled in the art of the present technique are appreciated that unless otherwise defined, all terms used herein（Including technical term and scientific terminology）With with art of the present invention in those of ordinary skill general understanding identical meaning.It should also be understood that those terms defined in such as general dictionary should be understood that with the meaning consistent with the meaning in the context of prior art, and unless defined as here, will not be with idealizing or excessively formal implication be explained.
For ease of the understanding to the embodiment of the present invention, embodiments of the present invention are described below in detail, are exemplary by reference to the implementation method of Description of Drawings, be only used for explaining the present invention, and be not construed as limiting the claims.
Fig. 2 is the date storage method schematic flow sheet based on point table of first embodiment of the invention.
As shown in Fig. 2 the date storage method based on point table of present embodiment, comprises the following steps：
Step S1, in storing data in master data table by the data genaration date.
Preferably, this step creates master data table using list data structures.List is a kind of linear list in STL, it is characterized in quickly to increase or delete element in any position, using the date where list data structure storages data to be transferred, insertion behaviour can be carried out in tail of the queue, such that it is able to perform data addition operation in the tail of the queue of master data table；And traveled through from head of the queue, shifted in batches by date with when data are shifted.
Data are stored respectively by month, the naming rule of tables of data represents the time for tables of data basic name+YYYY+MM, wherein Y, and M represents month, such as invinfo201506.For invoice data, the key element that data record is included mainly has：Serial number（It is globally unique, identify this record）, the date of making out an invoice, invoicing time, the amount of money of making out an invoice etc..Most of query statistic work are all based on serial number inquiry or date-time section of making out an invoice is counted.
Data in the master data table are polled by step S2, when in the presence of pre-conditioned data are met, log-on data point table process.
Here an independent data point table thread can perform specifically point table process since poll, after the completion of poll.
Preferably, poll can be inquired about with the presence or absence of the data for meeting jump condition since the head of the queue of master data table.Here jump condition, can preset, it is generally the case that described pre-conditioned for data message is complete, the date of formation of data meets a point table requirement.For invoice data, it refers to just to meet rule to meet jump condition, and such as invoice data key element is complete, the date meets transfer requirement and transfer flag bit is 0.If it does not exist, then continuing poll；If it is present screen qualified data and by date or month packet arrangement, and extract all dates and be stored in list data structures.Traversal date list, can obtain a point table name according to current days value first, such as basic table name+YYYY+MM.
Step S3, the data of log-on data point table process is transferred in corresponding data point table, and delete the corresponding data in master data table.
Before this step, after poll is completed, can also include, the data are grouped arrangement by date, and extract all dates and be stored in list data structures；Traversal date list, the current date according to the data obtains point table name in the current date affiliated month；Check the database with the presence or absence of point table corresponding with described point of table name；When in the absence of the corresponding point of table, point table according to described point of table name is created.
When there is the corresponding point of table, the data transfer flag bit in master data table corresponding to the month is updated to 1, during the data in the month are inserted into corresponding point of table, while shifting the data that flag bit is 1 in deleting master data table.Whether transfer flag bit here, be transferred for flag data, and transfer flag bit default value is 0, and expression is not yet shifted, it is preferred that carry out automatic assignment when data insert master data table.
This method can also include：
Step S4, sets first field and second field on record point table maximum date on record point table minimum date in database；After data point table thread is once divided table, first field and the second field are updated using the minimum date in date list and minimum date；Time limit storage thread obtains the date of the current list in database, and the data time limit maximum virtual value according to data lab setting calculates the earliest date for needing to preserve data；Delete the data point table between the earliest date and minimum date.
After the system operation long period, memory data output is than larger, on the one hand there is considerable influence for the speed inquired about, on the other hand because system memory size is limited, it is impossible to unconfined interpolation data always, for the system without particular/special requirement, the general data storage for preserving 3 years, therefore the method for this programme design is that the date of the earliest data that must be preserved is calculated according to current date, and obtains the name of all tables of data before the date, and carries out deletion action one by one.
In this step, independent time limit storage thread can be set, for carrying out time limit storage operation.It is used for recording the minimum days and maximum days of point table in database using two fields, the minimum days of point table is such as represented with MinDate, the maximum days of point table is represented with MaxDate.After data point table thread is once divided table, just go to update two field values with the maximum date in date list and minimum date.
Time limit storage thread obtains current system days value first, the maximum data virtual value for then being set according to system（Such as 3 years）, the days value DelDate of the earliest data for needing to preserve can be calculated, then delete the data point table between MinDate and DelDate.
Present embodiment is applied to inquiry and the statistic processes of data, can be carried out as follows：
For inquiry operation, GUID serial number typically according to certain data carries out inquiry operation, to this, traversal queries are carried out by nearby principle, that is, is first looked in base table with the presence or absence of the corresponding data record of the serial number, then nearby principle by date, the time period inquired about as needed, the scope in time and month where calculating, so as to be inferred to the name of tables of data where it, then travels through all of data in the table and is inquired about.
For counting the data manipulation in certain time period, method is the scope for calculating days according to the time period first, the time period for counting as needed, the scope in time and month where calculating, the scope of Score Lists can be obtained, then counted by required condition in base table and this point of table, can so avoid traveling through all of point of table, improved the response speed of operation.
Need exist for explanation be, it is two different threads concurrently performing because query statistic operates thread and time limit storage thread, in the presence of the possibility for operating same tables of data simultaneously, therefore need to carry out synchronous working between two threads, prevent from having deleted tables of data on one side, and another side is carrying out the same inquiry operation of table.Therefore, create a mutex CMutex, either query statistic thread or time limit store thread, it is necessary to get CMutex first before peration data table；After end of operation, CMutex is discharged immediately.Obtain and the code between release CMutex is tried one's best and accomplishes most to simplify, that is, reduce the holding time of CMutex as far as possible.
Fig. 3 is the date storage method flow chart based on point table of first embodiment of the invention.
As shown in figure 3, the date storage method based on point table of the present embodiment, comprises the following steps：
Step S201, inquiry is with the presence or absence of the data for meeting jump condition.When there are qualified data, step S202 is performed；When in the absence of qualified data, then repeat step S201, inquires about next data.
The present embodiment is inquired about data first, does not emphasize to call master data table here, and data can be stored in master data table, but this is not required in that condition, or a single data.Simply, in the presence of master data table, especially when master data table is built in the way of list data structures, the data genaration date is stored data in master data table, is more beneficial for the batch transfer of data.
Step S202, obtains date range list where data.
Step S203, travels through the date, and inquiry whether there is corresponding point of table.Here traversal, can begin stepping through in the data of date list from head of the queue.When there is corresponding point of table, step S205 is performed；When in the absence of corresponding point of table, step S204 is performed.
Fig. 3 show in the present embodiment the process schematic that by list data structures the data on corresponding date are performed with addition and polling operation.Here poll, since head of the queue.And to the addition of acquired new date data after having traveled through, can be since tail of the queue simultaneously, in this manner it is ensured that the traversal since head of the queue and the addition since tail of the queue, can be carried out, without influencing each other or conflicting.
The data of log-on data point table thread are transferred in corresponding data point table, and delete the corresponding data in master data table.
Step S204, creates of that month point of table.
Step S205, updates the flag bit of corresponding data record in underlying table.
Step S206, point table of copy corresponding data to correspondence month.
Step S207, deletes corresponding data in underlying table.Here underlying table is master data table.Next operation is then performed, step S201 is returned to.
The date storage method based on point table of the present embodiment, in storing data in master data table by the data genaration date first, then the data in the master data table are polled, when in the presence of pre-conditioned data are met, log-on data point table thread, qualified data are transferred in corresponding data point table, and delete the corresponding data in master data table；Furthermore it is also possible to calculate the date of the earliest data that must be preserved according to current date, and the name of all tables of data before the date is obtained, and carry out deletion action one by one, so as in the case where memory capacity need not be increased, meet the amount of storage demand of user.After being stored to data according to storage method of the invention, one amount of storage of tables of data drops to 100,000 grades for even hundred million grades from millions, inquiry and Statistical Speed are obviously improved, the query statistic stand-by period dropped to second level from tens seconds, the demand of user can effectively be met, Consumer's Experience is lifted.
Second embodiment
Present embodiment is a kind of data-storage system based on point table.Fig. 5 is the structural representation of the data-storage system based on point table of second embodiment of the invention.
As shown in figure 5, the data-storage system based on point table of present embodiment, including：Tables of data creating unit 1, preliminary memory cell 2, poll units 3, point table unit 4, deletion unit 5；Wherein,
The tables of data creating unit 1 is used to create master data table；
The preliminary memory cell 2 is connected with tables of data creating unit 1, in storing data in master data table by the data genaration date；
The poll units 3 are used to be polled the data in the master data table, when in the presence of pre-conditioned data are met, log-on data point table process；
Preferably, here described pre-conditioned for data message is complete, the date of formation of data meets a point table requirement.
Described point of table unit 4 is used to be transferred to the data of log-on data point table process in corresponding data point table；
The deletion unit 5 is connected with described point of table unit and tables of data creating unit, and the corresponding data of the complete component list of table unit is being divided for deleting in master data table.
The system also includes：Threshold value date setting/updating block 6, data time limit computing unit 7；Wherein,
The threshold value date setting/updating block 6 is used to be set in database first field and second field on record point table maximum date on record point table minimum date；It is additionally operable to after data point table thread is once divided table, first field and the second field be updated using the minimum date in date list and minimum date；
The data time limit computing unit 7 is used to obtain the date of the current list in database, and the data time limit maximum virtual value according to data lab setting calculates the earliest date for needing to preserve data；
The deletion unit 5 is also connected with the threshold value date setting/updating block 6 and the data time limit computing unit 7, for deleting the data point table between the earliest date and minimum date.
Preferably, the preliminary memory cell 2 is further used for creating master data table using list data structures, and data addition operation is performed in the tail of the queue of master data table；The poll units 3 are further used for performing poll since the head of the queue of master data table.
Preferably, described point of table unit includes：Packet subelement, an extraction point table name subelement, an inquiry point table subelement, flag bit update subelement, an execution point table subelement；Wherein,
The packet subelement is used to for the data to be grouped arrangement by date, and extracts all dates and be stored in list data structures；The extraction point table name subelement is used to travel through date list, and the current date according to the data obtains point table name in the current date affiliated month；The inquiry point table subelement is used to check the database with the presence or absence of point table corresponding with described point of table name；The tables of data creating unit 1 is additionally operable to when in the absence of the corresponding point of table, point table according to described point of table name is created；The flag bit updates subelement to be used to, when there is the corresponding point of table, the data transfer flag bit in master data table corresponding to the month is updated into 1；In performing point table subelement and being used to for the data in the month to insert corresponding point of table；Unit 3 is deleted to be additionally operable to shift the data that flag bit is 1 in deletion master data table.
The data-storage system based on point table of present embodiment, has the advantages that：
1. query statistic speed is significantly improved：Due to data monthly being preserved, an amount of storage for tables of data drops to 100,000 grades for even hundred million grades from millions, and inquiry velocity is obviously improved, after tested, before point table it is undesirable in the case of, the query statistic stand-by period can reach tens seconds, the limit that seriously be may wait for beyond user；After point table it is undesirable in the case of, the query statistic stand-by period can reach second level, can effectively meet the demand of user.
2. memory space utilization rate is improved：In the case where time limit storage is not carried out, the impossible unconfined addition for carrying out data of general data server memory capacity, and hard-disk capacity can not possibly unconfined expansion.Therefore after carrying out time limit storage, the application demand of user can in the case where memory capacity need not be increased, be met.
As seen through the above description of the embodiments, those skilled in the art can be understood that the present invention can add the mode of required general hardware platform to realize by software.Based on such understanding, the part that technical scheme substantially contributes to prior art in other words can be embodied in the form of software product, the computer software product can be stored in storage medium, such as ROM/RAM, magnetic disc, CD, including some instructions are used to so that a computer equipment（Can be personal computer, server, or network equipment etc.）Perform the method described in some parts of each embodiment of the invention or embodiment.
Each embodiment in this specification is described by the way of progressive, and identical similar part is mutually referring to what each embodiment was stressed is the difference with other embodiment between each embodiment.For especially for device or system embodiment, because it is substantially similar to embodiment of the method, so describing fairly simple, the relevent part can refer to the partial explaination of embodiments of method.Apparatus and system embodiment described above is only schematical, the wherein described unit illustrated as separating component can be or may not be physically separate, the part shown as unit can be or may not be physical location, a place is may be located at, or can also be distributed on multiple NEs.Some or all of module therein can be according to the actual needs selected to realize the purpose of this embodiment scheme.Those of ordinary skill in the art are without creative efforts, you can to understand and implement.
The above; the only present invention preferably specific embodiment, but protection scope of the present invention is not limited thereto, any one skilled in the art the invention discloses technical scope in; the change or replacement that can be readily occurred in, should all be included within the scope of the present invention.Therefore, protection scope of the present invention should be defined by scope of the claims.
Claims (10)
1. it is a kind of based on the date storage method for dividing table, it is characterised in that methods described comprises the following steps：
In master data table being stored data in by the data genaration date；
Data in the master data table are polled, when in the presence of pre-conditioned data are met, log-on data point table thread；
The data of log-on data point table thread are transferred in corresponding data point table, and delete the corresponding data in master data table.
2. date storage method according to claim 1, it is characterised in that
Described pre-conditioned for data message is complete, the date of formation of data meets a point table requirement；
The data point table is arranged by month.
3. date storage method according to claim 2, it is characterised in that methods described also includes：
First field and second field on record point table maximum date on record point table minimum date are set in database；After data point table thread is once divided table, first field and the second field are updated using the minimum date in date list and minimum date；
Time limit storage thread obtains the date of the current list in database, and the data time limit maximum virtual value according to data lab setting calculates the earliest date for needing to preserve data；
Delete the data point table between the earliest date and minimum date.
4. date storage method according to claim 1, it is characterised in that described by the data genaration date to store data in master data table, further for：Master data table is created using list data structures, data addition operation is performed in the tail of the queue of master data table；
The data in the master data table are polled, further for：Poll is performed since the head of the queue of master data table.
5. date storage method according to claim 4, it is characterised in that the data by log-on data point table thread are transferred in corresponding data point table, further for：
The data are grouped arrangement by date, and extract all dates and be stored in list data structures；
Traversal date list, the current date according to the data obtains point table name in the current date affiliated month；
Check the database with the presence or absence of point table corresponding with described point of table name；
When in the absence of the corresponding point of table, point table according to described point of table name is created；
When there is the corresponding point of table, the data transfer flag bit in master data table corresponding to the month is updated to 1, during the data in the month are inserted into corresponding point of table, while shifting the data that flag bit is 1 in deleting master data table.
6. it is a kind of based on the data-storage system for dividing table, it is characterised in that the system includes：Tables of data creating unit, preliminary memory cell, poll units, point table unit, deletion unit；Wherein,
The tables of data creating unit is used to create master data table；
The preliminary memory cell is connected with tables of data creating unit, in storing data in master data table by the data genaration date；
The poll units are used to be polled the data in the master data table, when in the presence of pre-conditioned data are met, log-on data point table thread；
Described point of table unit is used to be transferred to the data of log-on data point table thread in corresponding data point table；
The deletion unit is connected with described point of table unit and tables of data creating unit, and the corresponding data of the complete component list of table unit is being divided for deleting in master data table.
7. data-storage system according to claim 6, it is characterised in that
Described pre-conditioned for data message is complete, the date of formation of data meets a point table requirement；
Described point of table unit is further used for being arranged the data point table by month.
8. data-storage system according to claim 7, it is characterised in that the system also includes：Threshold value date setting/updating block, data time limit computing unit；Wherein,
The threshold value date setting/updating block is used to be set in database first field and second field on record point table maximum date on record point table minimum date；It is additionally operable to after data point table thread is once divided table, first field and the second field be updated using the minimum date in date list and minimum date；
The data time limit computing unit is used to obtain the date of the current list in database, and the data time limit maximum virtual value according to data lab setting calculates the earliest date for needing to preserve data；
The deletion unit is also connected with the threshold value date setting/updating block and the data time limit computing unit, for deleting the data point table between the earliest date and minimum date.
9. data-storage system according to claim 6, it is characterised in that
The preliminary memory cell is further used for creating master data table using list data structures, and data addition operation is performed in the tail of the queue of master data table；
The poll units are further used for performing poll since the head of the queue of master data table.
10. data-storage system according to claim 9, it is characterised in that described point of table unit includes：Packet subelement, an extraction point table name subelement, an inquiry point table subelement, flag bit update subelement, an execution point table subelement；Wherein,
The packet subelement is used to for the data to be grouped arrangement by date, and extracts all dates and be stored in list data structures；
The extraction point table name subelement is used to travel through date list, and the current date according to the data obtains point table name in the current date affiliated month；
The inquiry point table subelement is used to check the database with the presence or absence of point table corresponding with described point of table name；
The tables of data creating unit is additionally operable to when in the absence of the corresponding point of table, point table according to described point of table name is created；
The flag bit updates subelement to be used to, when there is the corresponding point of table, the data transfer flag bit in master data table corresponding to the month is updated into 1；
In performing point table subelement and being used to for the data in the month to insert corresponding point of table；
Unit is deleted to be additionally operable to shift the data that flag bit is 1 in deletion master data table.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201511018398.2A CN106933836B (en) | 2015-12-29 | 2015-12-29 | Data storage method and system based on sub-tables |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201511018398.2A CN106933836B (en) | 2015-12-29 | 2015-12-29 | Data storage method and system based on sub-tables |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106933836A true CN106933836A (en) | 2017-07-07 |
| CN106933836B CN106933836B (en) | 2021-07-06 |
Family
ID=59442305
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201511018398.2A Active CN106933836B (en) | 2015-12-29 | 2015-12-29 | Data storage method and system based on sub-tables |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106933836B (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107657014A (en) * | 2017-09-26 | 2018-02-02 | 上海斐讯数据通信技术有限公司 | A kind of dynamic of Large Volume Data table divides table method and system |
| CN109285080A (en) * | 2018-09-04 | 2019-01-29 | 中国平安财产保险股份有限公司 | Policy information scroll processing method, device, computer equipment and storage medium |
| CN110019436A (en) * | 2017-07-14 | 2019-07-16 | 北京国双科技有限公司 | Data import deriving method and device, tables of data treating method and apparatus |
| CN110795431A (en) * | 2019-10-28 | 2020-02-14 | 天津同阳科技发展有限公司 | Environment monitoring data processing method, device, equipment and storage medium |
| CN112214503A (en) * | 2020-10-10 | 2021-01-12 | 深圳壹账通智能科技有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN112395270A (en) * | 2020-11-27 | 2021-02-23 | 网络通信与安全紫金山实验室 | Data management method, device, equipment and storage medium |
| CN112527828A (en) * | 2020-12-10 | 2021-03-19 | 福建新大陆支付技术有限公司 | Tax control record storage method and retrieval query method for tax control machine |
| CN115455015A (en) * | 2022-08-08 | 2022-12-09 | 中亿(深圳)信息科技有限公司 | Mass data storage method and device and storage medium |
| CN116775648A (en) * | 2023-06-05 | 2023-09-19 | 广州慧云网络科技有限公司 | Data processing method, device, equipment and medium of machine room monitoring equipment |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102073697A (en) * | 2010-12-28 | 2011-05-25 | 中兴通讯股份有限公司 | Data processing method and data processing device |
| US20150278310A1 (en) * | 2014-03-26 | 2015-10-01 | Nec Corporation | Database device |
-
2015
- 2015-12-29 CN CN201511018398.2A patent/CN106933836B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102073697A (en) * | 2010-12-28 | 2011-05-25 | 中兴通讯股份有限公司 | Data processing method and data processing device |
| US20150278310A1 (en) * | 2014-03-26 | 2015-10-01 | Nec Corporation | Database device |
Non-Patent Citations (2)
| Title |
|---|
| 王亚玲 等: "数据库系统应用分片中间件", 《计算机系统应用》 * |
| 袁震: "基于消息中间件的数据同步更新方法", 《兵工自动化》 * |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110019436A (en) * | 2017-07-14 | 2019-07-16 | 北京国双科技有限公司 | Data import deriving method and device, tables of data treating method and apparatus |
| CN110019436B (en) * | 2017-07-14 | 2021-04-09 | 北京国双科技有限公司 | Data import \ export method and device and data table processing method and device |
| CN107657014A (en) * | 2017-09-26 | 2018-02-02 | 上海斐讯数据通信技术有限公司 | A kind of dynamic of Large Volume Data table divides table method and system |
| CN109285080A (en) * | 2018-09-04 | 2019-01-29 | 中国平安财产保险股份有限公司 | Policy information scroll processing method, device, computer equipment and storage medium |
| CN109285080B (en) * | 2018-09-04 | 2023-04-04 | 中国平安财产保险股份有限公司 | Policy information rolling processing method and device, computer equipment and storage medium |
| CN110795431A (en) * | 2019-10-28 | 2020-02-14 | 天津同阳科技发展有限公司 | Environment monitoring data processing method, device, equipment and storage medium |
| CN112214503A (en) * | 2020-10-10 | 2021-01-12 | 深圳壹账通智能科技有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN112395270A (en) * | 2020-11-27 | 2021-02-23 | 网络通信与安全紫金山实验室 | Data management method, device, equipment and storage medium |
| CN112527828A (en) * | 2020-12-10 | 2021-03-19 | 福建新大陆支付技术有限公司 | Tax control record storage method and retrieval query method for tax control machine |
| CN115455015A (en) * | 2022-08-08 | 2022-12-09 | 中亿(深圳)信息科技有限公司 | Mass data storage method and device and storage medium |
| CN115455015B (en) * | 2022-08-08 | 2024-01-26 | 中亿(深圳)信息科技有限公司 | Mass data storage method and device and storage medium |
| CN116775648A (en) * | 2023-06-05 | 2023-09-19 | 广州慧云网络科技有限公司 | Data processing method, device, equipment and medium of machine room monitoring equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106933836B (en) | 2021-07-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106933836A (en) | A kind of date storage method and system based on point table | |
| CN111460023B (en) | Method, device, equipment and storage medium for processing service data based on elastic search | |
| CN101876983B (en) | Method for partitioning database and system thereof | |
| CN107818115B (en) | Method and device for processing data table | |
| CN111459985B (en) | Identification information processing method and device | |
| EP2924594A1 (en) | Data encoding and corresponding data structure in a column-store database | |
| CN102750317B (en) | Method and device for data persistence processing and data base system | |
| CN103678392A (en) | Data increment and merging method and device for achieving method | |
| CN104317928A (en) | Service ETL (extraction-transformation-loading) method and service ETL system both based on distributed database | |
| CN111046034A (en) | Method and system for managing memory data and maintaining data in memory | |
| CN103678556A (en) | Method for processing column-oriented database and processing equipment | |
| CN103370691A (en) | Managing buffer overflow conditions | |
| CN102054000B (en) | Data querying method, device and system | |
| EP2380090B1 (en) | Data integrity in a database environment through background synchronization | |
| CN103544153A (en) | Data updating method and system based on database | |
| CN110727727B (en) | Statistical method and device for database | |
| CN102780603B (en) | Web traffic control method and device | |
| CN101739292A (en) | Application characteristic-based isomeric group operation self-adapting dispatching method and system | |
| JP2003296171A (en) | Electronic business form management method and program | |
| CN110096509A (en) | Realize that historical data draws the system and method for storage of linked list modeling processing under big data environment | |
| CN103793493A (en) | Method and system for processing car-mounted terminal mass data | |
| CN105808653A (en) | User label system-based data processing method and device | |
| CN104346458A (en) | Data storage method and device | |
| CN104317957A (en) | Open platform and system for processing reports and report processing method | |
| CN105787058A (en) | User label system and data pushing system based on same |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |