CN105608208A - System and method for storing big data concurrently - Google Patents
System and method for storing big data concurrently Download PDFInfo
- Publication number
- CN105608208A CN105608208A CN201511004846.3A CN201511004846A CN105608208A CN 105608208 A CN105608208 A CN 105608208A CN 201511004846 A CN201511004846 A CN 201511004846A CN 105608208 A CN105608208 A CN 105608208A
- Authority
- CN
- China
- Prior art keywords
- data
- sequence number
- list
- concurrent
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
Abstract
The invention relates to the field of database storage, in particular to a system and a method for storing big data concurrently. The system comprises a configuring unit, an obtaining module, a route configuring module, a determining module and a route module, wherein the configuring unit sets identifier rules and determination rules during data distribution; the obtaining module obtains a concurrent number serial number and a sequence serial number of each datum based on the identifier rules; the route configuring module sets data distribution determination conditions and distribution storage forms based on the distribution determination rules; the determining module performs distribution determination on each datum and determines the form serial numbers of the data; the route module searches for storage routes corresponding to the form serial numbers based the determination conditions of the route configuring module and stores each datum in a corresponding form module. The system can store thousands of data each second to the maximum by setting data serial numbers with time dimensions as the reference, therefore, rapid storage of millions or ten millions of data can be achieved, and the risk of storage system crashes in an overload storage process is reduced.
Description
Technical field
The present invention relates to a kind of database field of storage, relate in particular to a kind of large Data Concurrent storageSystem and method.
Background technology
Along with the development of internet, many services all enter procedure, as E-business service,Large data analysis service, logistics service, communication service class, the service of third party's network interface etc.,Very because the development of network makes these services in the business datum amount all the time producingHuge, and need to store in time. Due to the storage speed of data in storing processThe speed producing with respect to data is slow, thereby makes storage system constantly carry out excess loadIn storing process, there is the risk of collapse, the service effectiveness of the whole server of meeting remote-effects.
Summary of the invention
The object of the invention is to, a kind of large Data Concurrent storage system is provided, for solve withUpper technical problem; The present invention also provides a kind of large Data Concurrent storage means.
The present invention realizes by the following technical solutions:
A kind of large Data Concurrent storage system, is applied to the instant some numbers that produce of service endAccording to storage in, comprising:
Dispensing unit, for arranging the indications of every described data in the time carrying out data distributionThe decision rule of rule and shunting, and described indications comprises generic sequence number, number of concurrent sequenceNumber and list sequence number;
Acquisition module, is connected with described dispensing unit and described service end respectively, described in basisNumber of concurrent sequence number and the generic sequence number of every described data of Rule of indications;
Path configurations module, is connected with described dispensing unit, to advise according to the judgement of described shuntingThe decision condition of data distribution and the storage list of shunting are set;
Determination module, is connected with described dispensing unit and described acquisition module respectively, with according to instituteState number of concurrent sequence number and utilize described decision rule to shunt judgement to data every described, sentenceThe list sequence number of fixed described data;
Several list modules, for storing the data of each list;
Path module, respectively with described determination module, path configurations module and described several tablesSingle module connects, and searches described list sequence number according to the decision condition of described path configurations moduleCorresponding store path, and described in every data being stored to accordingly according to described store pathIn list module.
Preferably, above-mentioned large Data Concurrent storage system also comprises:
The major control data center module being connected with described several list modules, if for by describedDry form data merging is stored as summary sheet, and the data in described summary sheet are synchronized to masterIn control database.
The application also provides a kind of large Data Concurrent storage means, can be applicable to service endIn the storage of instant some data that produce, described method comprises:
Step 1, arrange the indications of every described data in the time carrying out data distribution rule andThe decision rule of shunting, and the judgement bar of data distribution is set according to the decision rule of described shuntingThe storage list of part and shunting; Described indications comprises generic sequence number, number of concurrent sequence numberWith list sequence number;
Step 2, according to the number of concurrent sequence of every described data of Rule of described indicationsNumber and generic sequence number;
Step 3, judges every number according to the storage list of the decision condition of described shunting and shuntingAccording to list sequence number;
Step 4, searches store path corresponding to described list sequence number according to described decision condition,And according to described store path, every data are stored in corresponding described list module.
Above-mentioned large Data Concurrent storage means also comprises:
Form data is merged and is stored as summary sheet, and the data in described summary sheet are synchronized toIn major control data storehouse.
In above-mentioned large Data Concurrent storage means:
Described number of concurrent sequence number refers to that, in a period of time of current appointment, the data that store are pressedAccording to the sequence number of natural numerical order column-generation.
In above-mentioned large Data Concurrent storage means:
Described number of concurrent sequence number represents with the decimal number of 4, described number of concurrent sequence number from0000 starts counting or counts since 0001.
In above-mentioned large Data Concurrent storage means:
The decision condition of described data distribution refers to identical number that every list storage setsData.
In above-mentioned large Data Concurrent storage means:
Described list sequence number adopts the decimal system or the hexadecimal mode of 2 to count, described inList sequence number is since 00 counting or since 01 counting.
In above-mentioned large Data Concurrent storage means:
Described generic sequence number adopts the mode taking time dimension as benchmark to represent: with 16YYYYMMDDHHMMSS represents, wherein, the YYYY expression of years, MM represents month, DD representsDay, HH represents hour, and MM represents minute, and SS represents second.
The present invention is by data serial number taking time dimension as benchmark is set, each in second maximumThe data volume that can reach 1,000 of storages, can realize depositing fast of 1,000,000 grades of millions data volumesStorage, reduces the risk that storage system is collapsed in excess load storing process, is particularly useful for electronicsBusiness service, large data analysis service, logistics service, communication service class, third party's network connectThe storage of the information such as sequence information data, communication information data in oral business.
Brief description of the drawings
Fig. 1 is the schematic flow sheet of a kind of large Data Concurrent storage means of invention;
Fig. 2 is the structural representation of a kind of large Data Concurrent storage system of the present invention.
Detailed description of the invention
Below in conjunction with the accompanying drawing in the embodiment of the present invention, to the technical side in the embodiment of the present inventionCase is clearly and completely described, and obviously, described embodiment is only one of the present inventionDivide embodiment, instead of whole embodiment. Based on the embodiment in the present invention, this area is generalThe every other enforcement that logical technical staff obtains under the prerequisite of not making creative workExample, all belongs to the scope of protection of the invention.
It should be noted that, in the situation that not conflicting, the embodiment in the present invention and embodimentIn feature can mutually combine.
Below in conjunction with the drawings and specific embodiments, the invention will be further described, but not as thisThe restriction of invention.
As shown in Figure 1, a kind of large Data Concurrent storage means, a large amount of for to instant generationData are stored. Before storage data, can add corresponding stream in the beginning of every dataWater number, prevents the order entanglement of save data. It comprises generic sequence number, number of concurrent sequence numberWith list sequence number, three is combined to the serial number that just forms corresponding data, namely markShow symbol.
It is pointed out that described generic sequence number is not subject to the restriction of figure place, can adopt baseIn the year of time parameter YYYY, month MM, day DD, time HH, point MM, second SS form rawBecome the time series YYYYMMDDHHMMSS of 16, also can adopt based on year YY, month MM,Day DD, time HH, point MM generate the time series YYMMDDHHMM of 12; Certainly also canTo adopt the mode that represents of other Arabic numerals and/or time, do not repeat one by one at this.
Number of concurrent sequence number refers in a period of time of current appointment, the data that store according to fromThe sequence number that so Number Sequence sequence generates, the figure place of this sequence number is unrestricted. This sequence number canWith since 0 counting, also can count since 1. After this is one second, number of concurrent sequenceNumber make zero, restart counting.
In embodiments of the invention, a period of time of appointment is one second or 1000 milliseconds, arrangesThe maximum amount of data that can receive in these 1000 milliseconds of times is 1000, and since 1 counting,It is 4 figure places that concurrent sequence number is set simultaneously. The time is 15 o'clock on the 13rd November in 2015In in 21 points of 46 seconds one seconds, acceptable maximum amount of data is 1000, within this second,The 20th article of time series that data are corresponding number receiving is 20151113152146, number of concurrentSequence number is 0020; In the time that the Article 1 data of next second produce, corresponding time series numberBe 20151113152147, number of concurrent sequence number is 0001.
List sequence number is the sequence number of the list of storage data, as the mark of submeter or pagingKnow. List sequence number can correspond to different data objects, for example, and in trading order form dataIn information, each sequence number can be expressed as the list of different suppliers' storage data, alsoCan be expressed as multiple for storing the list of data; In logistics order data information, canThe logistics information data that initiate corresponding different regions etc. In a word, each list sequence number correspondenceThe list of storage data.
Similarly, the expression of this list sequence number is not subject to the restriction of figure place, can adopt the decimal systemRepresent, also can adopt hexadecimal number to represent. In the time adopting two decimal numbers to represent, tableSingle sequence number can 99 tables of from 01 to 99 expression; In the time adopting hexadecimal representation,The sequence number of list can represent 165 lists from 01 to FF. Similarly, also can be from00 starts the sequence number as list. Therefore, the time is 15 o'clock on the 13rd November in 201521 points 46 seconds, number of concurrent sequence number is 0020, the serial number that is stored in the 88th table is passableBe expressed as 20151113152146002088.
Before storage data, the decision condition of shunting need to be set: set whenever producing C barWhen data, just once shunt, every list is namely set can only stores the number of dataC. For example set whenever produce 20 data just shunt once, as serial number from20151113152146000101-20151113152146002001, represents when time series to be20151113152146, and the number of concurrent sequence number of the data that produce is between 0001-0020Data, be saved in first list 01; Serial number from20151113152146002102-2015111315214604002, represents when time series to be20151113152146, and the number of concurrent sequence number of the data that produce is between 0021-0040Data, be saved in second list 02. As produced 1000 data when per second,While once shunting according to every 20 data, the list number that needs to store data is for extremelyFew 50. Below be also the data storage that obtains required storage according to the decision rule of shuntingTo the method for corresponding list sequence number.
The application's the large Data Concurrent storage means of one comprises the steps:
Step 1, arranges rule and the shunting of the indications of every data in the time carrying out data distributionDecision rule, and according to shunting decision rule arrange data distribution decision condition and pointThe storage list of stream; Indications comprises generic sequence number, number of concurrent sequence number and list sequence number;
Step 2, according to the number of concurrent sequence number of every data of Rule of indications and orderSequence number;
Step 3, judges every data according to the decision condition of shunting and the storage list of shuntingList sequence number;
Step 4, according to decision condition look-up table simple sequence number corresponding store path, and according toStore path is stored to every data in corresponding list module.
Wherein, list comprises single table and summary sheet, and single table refers to stores the table of setting bar logarithmic dataSingle, summary sheet refers to the data of each single table is gathered, generates a summary table, after facilitatingContinuous statistics and data query, when list sequence number is during from 01 open numbering, summary sheet can makeWith 00 as its list sequence number.
In addition, the data of summary sheet can also be stored to major control data center, on realization is synchronousPass. Can be according to the temporal frequency of setting, for example every day or weekly frequency, will showData in list merge the major control data center that is stored to; Also can work as the data volume of preserving and reach oneWhen individual default value, synchronously upload; Certainly, also can irregularly obtain uploading data extremelyMajor control data center. It is convenient that data are synchronized in major control data center or other types databaseStatistical analysis or inquiry etc.
As shown in Figure 2, a kind of large Data Concurrent storage system, is applied to service end is produced immediatelyIn the storage of some raw data, wherein, dispensing unit for arrange indications rule andThe decision rule of data distribution. Indications comprises generic sequence number, number of concurrent sequence number and listSequence number, storage when data, is added into this indications on the head of every data, as dataSerial number, deposits in corresponding list together. The rule that indications is set namely arranges signThe concrete expression mode of symbol, in the present embodiment, adopts the data that are benchmark based on time dimensionDivergence Accordance, as adopt the generic sequence being formed by the time number of 14,4 concurrent number sequencesRow number and the list sequence number of 2.
Acquisition module is connected with described dispensing unit and described service end respectively, for obtaining dataNumber of concurrent information, generate the number of concurrent sequence number of 4 figure places. The time of hypothesis is 2015A certain the data of obtaining when 15: 21: 46 on the 13rd November, its generic sequence numberBe 20151113152146. Number of concurrent sequence number refers to the data volume producing into Millisecond, whenWhen the data volume of accepting in one second is 1000, the number of concurrent sequence number of the 20th article of data isArticle 0020, the 1000, the number of concurrent sequence number of data is 1000.
Path configurations module is connected with dispensing unit, data to be set according to the decision rule of shuntingThe decision condition of shunting and the storage list of shunting. The decision condition of shunting is set, establishes exactlyWhen the fixed data whenever generation setting number, just once shunt, thus the storage of configuration shuntingList.
Path module connects with determination module, path configurations module and several list modules respectivelyConnect, according to the decision condition look-up table simple sequence number corresponding store path of path configurations module,And according to store path, every data are stored in corresponding list module. In addition also comprise,The major control data center module being connected with list module, for by the data message of each listBe synchronized in major control data storehouse.
The foregoing is only preferred embodiment of the present invention, not thereby limit enforcement of the present inventionMode and protection domain, to those skilled in the art, should recognize all utilizationsWhat description of the present invention and diagramatic content were made is equal to replacement and apparent variation gainedThe scheme arriving, all should be included in protection scope of the present invention.
Claims (9)
1. a large Data Concurrent storage system, is applied to instant some of producing of service endIn the storage of data, it is characterized in that, comprising:
Dispensing unit, for arranging the indications of every described data in the time carrying out data distributionThe decision rule of rule and shunting, and described indications comprises generic sequence number, number of concurrent sequenceNumber and list sequence number;
Acquisition module, is connected with described dispensing unit and described service end respectively, described in basisNumber of concurrent sequence number and the generic sequence number of every described data of Rule of indications;
Path configurations module, is connected with described dispensing unit, to advise according to the judgement of described shuntingThe decision condition of data distribution and the storage list of shunting are set;
Determination module, is connected with described dispensing unit and described acquisition module respectively, with according to instituteState number of concurrent sequence number and utilize described decision rule to shunt judgement to data every described, sentenceThe list sequence number of fixed described data;
Several list modules, for storing the data of each list;
Path module, respectively with described determination module, path configurations module and described several tablesSingle module connects, and searches described list sequence number according to the decision condition of described path configurations moduleCorresponding store path, and described in every data being stored to accordingly according to described store pathIn list module.
2. large Data Concurrent storage system according to claim 1, is characterized in that, alsoComprise: the major control data center module being connected with described several list modules, described in inciting somebody to actionSeveral form datas merging are stored as summary sheet, and the data in described summary sheet are synchronized toIn major control data storehouse.
3. a large Data Concurrent storage means, is characterized in that, is applied to service end instantIn the storage of some data that produce, described method comprises:
Step 1, arrange the indications of every described data in the time carrying out data distribution rule andThe decision rule of shunting, and the judgement bar of data distribution is set according to the decision rule of described shuntingThe storage list of part and shunting; Described indications comprises generic sequence number, number of concurrent sequence numberWith list sequence number;
Step 2, according to the number of concurrent sequence of every described data of Rule of described indicationsNumber and generic sequence number;
Step 3, judges every number according to the storage list of the decision condition of described shunting and shuntingAccording to list sequence number;
Step 4, searches store path corresponding to described list sequence number according to described decision condition,And according to described store path, every data are stored in corresponding described list module.
4. large Data Concurrent storage means according to claim 3, is characterized in that, alsoComprise: form data is merged and is stored as summary sheet, and by synchronous the data in described summary sheetIn major control data storehouse.
5. large Data Concurrent storage means according to claim 3, is characterized in that instituteState in a period of time that number of concurrent sequence number refers to current appointment, the data that store are according to natureThe sequence number that Number Sequence generates.
6. according to the large Data Concurrent storage means described in claim 3 or 5, it is characterized in that,Described number of concurrent sequence number represents with the decimal number of 4, and described number of concurrent sequence number is from 0000Start counting or count since 0001.
7. large Data Concurrent storage means according to claim 3, is characterized in that instituteThe decision condition of stating data distribution refers to the data of the identical number of every list storage setting.
8. according to the large Data Concurrent storage means described in claim 3 or 7, it is characterized in that,Described list sequence number adopts the decimal system or the hexadecimal mode of 2 to count, described listSequence number is since 00 counting or since 01 counting.
9. large Data Concurrent storage means according to claim 3, is characterized in that instituteTo state generic sequence number adopt the mode taking time dimension as benchmark to represent: with 16YYYYMMDDHHMMSS represents, wherein, the YYYY expression of years, MM represents month, DD representsDay, HH represents hour, and MM represents minute, and SS represents second.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201511004846.3A CN105608208B (en) | 2015-12-28 | 2015-12-28 | A kind of concurrent storage system of big data and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201511004846.3A CN105608208B (en) | 2015-12-28 | 2015-12-28 | A kind of concurrent storage system of big data and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105608208A true CN105608208A (en) | 2016-05-25 |
| CN105608208B CN105608208B (en) | 2019-04-02 |
Family
ID=55988147
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201511004846.3A Active CN105608208B (en) | 2015-12-28 | 2015-12-28 | A kind of concurrent storage system of big data and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105608208B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109299096A (en) * | 2018-09-25 | 2019-02-01 | 阿里巴巴集团控股有限公司 | A kind of processing method of pipelined data, device and equipment |
| CN110347513A (en) * | 2019-07-15 | 2019-10-18 | 中国工商银行股份有限公司 | Hot spot data lot size scheduling method and device |
| CN111930738A (en) * | 2020-01-16 | 2020-11-13 | 杭州隼目信息科技有限公司 | Intelligent shunting processing method for form data |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102209030A (en) * | 2011-05-19 | 2011-10-05 | 中兴通讯股份有限公司 | Service traffic splitting method, device and system |
| CN102402730A (en) * | 2010-09-15 | 2012-04-04 | 金蝶软件（中国）有限公司 | Dynamic business data distribution method and system |
| US20130223234A1 (en) * | 2012-02-29 | 2013-08-29 | International Business Machines Corporation | Multi-threaded packet processing |
| CN103778135A (en) * | 2012-10-18 | 2014-05-07 | 厦门雅迅网络股份有限公司 | Method for distribution storage and paging querying of real-time data |
| CN104850619A (en) * | 2015-05-15 | 2015-08-19 | 深圳市金蝶友商电子商务服务有限公司 | Receipt code generation method and apparatus |
| CN105072160A (en) * | 2015-07-17 | 2015-11-18 | 联动优势科技有限公司 | Serial number generating method and device, and a server |
-
2015
- 2015-12-28 CN CN201511004846.3A patent/CN105608208B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102402730A (en) * | 2010-09-15 | 2012-04-04 | 金蝶软件（中国）有限公司 | Dynamic business data distribution method and system |
| CN102209030A (en) * | 2011-05-19 | 2011-10-05 | 中兴通讯股份有限公司 | Service traffic splitting method, device and system |
| US20130223234A1 (en) * | 2012-02-29 | 2013-08-29 | International Business Machines Corporation | Multi-threaded packet processing |
| CN103778135A (en) * | 2012-10-18 | 2014-05-07 | 厦门雅迅网络股份有限公司 | Method for distribution storage and paging querying of real-time data |
| CN104850619A (en) * | 2015-05-15 | 2015-08-19 | 深圳市金蝶友商电子商务服务有限公司 | Receipt code generation method and apparatus |
| CN105072160A (en) * | 2015-07-17 | 2015-11-18 | 联动优势科技有限公司 | Serial number generating method and device, and a server |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109299096A (en) * | 2018-09-25 | 2019-02-01 | 阿里巴巴集团控股有限公司 | A kind of processing method of pipelined data, device and equipment |
| CN110347513A (en) * | 2019-07-15 | 2019-10-18 | 中国工商银行股份有限公司 | Hot spot data lot size scheduling method and device |
| CN111930738A (en) * | 2020-01-16 | 2020-11-13 | 杭州隼目信息科技有限公司 | Intelligent shunting processing method for form data |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105608208B (en) | 2019-04-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| AU2010265607B2 (en) | Method for finding, updating and synchronizing modified record item and data synchronizing device | |
| CN105608208A (en) | System and method for storing big data concurrently | |
| US10769132B1 (en) | Efficient storage and retrieval of time series data | |
| CN109873873A (en) | A kind of flight data delivery system, flight variation and message treatment method | |
| CN110740160B (en) | Multi-source data map gridding and data state real-time pushing system | |
| US10592827B2 (en) | Throttling solutions into a legacy inventory system during a service disruption | |
| CN107196848B (en) | Information push method and device | |
| CN105843933B (en) | The index establishing method of distributed memory columnar database | |
| CN108282508A (en) | Determination method and device, information-pushing method and the device in geographical location | |
| CN102880529A (en) | Memory data backup method and memory data backup system | |
| CN106878184A (en) | A kind of data message transmission method and device | |
| CN108062243A (en) | Generation method, task executing method and the device of executive plan | |
| CN109271449A (en) | A kind of distributed storage inquiry system file-based and querying method | |
| CN101610399A (en) | The method of the plan class service scheduling system and the class service dispatching that realizes a plan | |
| CN105338107A (en) | Stronghold operation synchronous management system and stronghold operation synchronous management method | |
| CN108092914B (en) | Network traffic load balancing scheduling method and device | |
| US20210216516A1 (en) | Management of a secondary vertex index for a graph | |
| CN103631975A (en) | Data extraction method and device | |
| CN106484714A (en) | A kind of storage method of behavior record and equipment | |
| CN101072156A (en) | Method and system for searching seed for P2P system | |
| CN116108094A (en) | Data integration method and device, electronic equipment and storage medium | |
| CN110225077A (en) | Synchronous method, device, computer equipment and the computer storage medium of change supply data | |
| CN108733728A (en) | Time series data statistical method, device, computer equipment and readable storage medium storing program for executing | |
| CN107291757B (en) | Pattern matching method and pattern matching device | |
| CN109002908A (en) | One kind is arranged an order according to class and grade dispatching method, system and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right |
Denomination of invention: A large data concurrent storage system and method Effective date of registration: 20210926 Granted publication date: 20190402 Pledgee: Bank of Communications Ltd. Shanghai Xuhui sub branch Pledgor: SHANGHAI HANDPAL INFORMATION TECHNOLOGY SERVICE Co.,Ltd. Registration number: Y2021310000079 |
|
| PE01 | Entry into force of the registration of the contract for pledge of patent right |