CN106844418A - A kind of internal storage data processing system - Google Patents
A kind of internal storage data processing system Download PDFInfo
- Publication number
- CN106844418A CN106844418A CN201611051186.9A CN201611051186A CN106844418A CN 106844418 A CN106844418 A CN 106844418A CN 201611051186 A CN201611051186 A CN 201611051186A CN 106844418 A CN106844418 A CN 106844418A
- Authority
- CN
- China
- Prior art keywords
- data
- storage
- creation module
- file
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2308—Concurrency control
- G06F16/2315—Optimistic concurrency control
- G06F16/2322—Optimistic concurrency control using timestamps
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
Abstract
The present invention relates to a kind of internal storage data processing system, the system includes：Communication module, for receiving various information and output result；Data creation module, the storage catalogue for determining data according to data class；Data are stored according to the form of data file, temporal information is included in the filename of the data file；Data read module, reads thread to extract data file from list successively for the information creating in data creation module to be multiple, reads the data in data creation module.The demand for solving to read and write database high concurrent by the present invention；And to the high efficiency storage of mass data and the demand for accessing；With the enhanced scalability and the demand of high availability to database.
Description
Technical field
The present invention relates to computer data field, more particularly to a kind of internal storage data processing system.
Background technology
Current memory database is shared by the way that the data in frequently-used data storehouse table in system are all mapped into main frame
In internal memory, each database table fixed is stored in shared drive by using a fixed array of data structures,
It is exactly that, using a memory block in internal memory an as table space for database table, the actual size according to token record stores this
Zoning is divided into multiple memory blocks to preserve the data in database table；Internal memory rope is set up in critical field in database table
Draw, indexed by the internal memory carries out real time access to critical data.Memory system is provided for the number in main memory database table
According to API API (the Application Programming Interface) interface modified and retrieve, application
Program when these database tables are accessed, by the data for calling the API of memory database to access in shared drive, rather than
The data in physical database table are directly accessed, therefore, it is possible to improve real time access performance of the system to critical data.
But, the table in relevant database is all to store some data structures for formatting, the group of each tuple field
Chengdu is the same, though it is not that each tuple is required for all of field, but database can distribute all of field for each tuple,
Such structure can be in order to the operation such as be attached between table and table, but it is also relational data from for another angle
One factor of storehouse performance bottleneck.It can not meet following " height " demand：To the demand of database high concurrent read-write；To magnanimity number
According to high efficiency storage and access demand；The demand of enhanced scalability and high availability to database.
The content of the invention
The technical problems to be solved by the invention are：Current data processing technique is difficult to solve to read database high concurrent
The demand write；And to the high efficiency storage of mass data and the demand for accessing；With the enhanced scalability and Gao Ke to database
With the demand of property.
To solve technical problem above, the invention provides a kind of internal storage data processing system, the system includes difference
Communication module, data operation modules and the data read module being connected with data creation module：
Communication module, for receiving various information and output result；
Data creation module, the data message for being received according to communication module determines the storage catalogue of data；By number
According to the form storage according to data file to storage server, temporal information is included in the filename of the data file；Institute
The storage server stated includes the disk array being made up of multiple disc drivers, carries out logic point to the disk array in advance
Area；
Data operation modules, for needing to data creation module in the operation in when adding token record, inquiry is described
One logic area of description information selection size matching, and will mark to record the memory block of free time in the logic area for writing selection
It is interior；
Data read module, for the multiple threads that read of the information creating in data creation module come successively from list
Middle extraction data file, reads the data in data creation module.
Further, after logical partition is carried out, multistage file directory is set up by setting rule in each logical partition；
The division rule of data file is set up, makes the size of the data file of storage in the range of setting；
During data storage, before storage server data storage, first analyze data is simultaneously true according to the foundation rule of multistage catalogue
Determine storage catalogue；
It is to set up index by the data storage file of setting time；
Storage server inquires about data using segmentation multi-thread concurrent technology come scanning index and the path of data file.
Further, described data creation module also includes：Determining unit, for determining data text in data creation module
The size of part record；Creating unit, for setting up the first memory block, the second memory block and the 3rd memory block in shared drive；
Storage unit, for the description information of database to be saved in into the first memory block, the description information of database includes each database
The description information of table.
Further, described data operation modules also include：Operating unit, the token record write-in choosing for addition will to be needed
In the logic area selected in idle memory block, and the position letter that token record is preserved in the second memory block.
Further, for data creation module, the foundation index is to set up multiple index.
Further, for data creation module, after index is set up, timing updates index.
Beneficial effect：Solve the demand to the read-write of database high concurrent；And to the storage of the high efficiency of mass data and
The demand of access；With the enhanced scalability and the demand of high availability to database, data storage and the speed for reading are improve.
Brief description of the drawings
Fig. 1 is a kind of synoptic diagram of internal storage data processing system of the invention.
Specific embodiment
Principle of the invention and feature are described below in conjunction with accompanying drawing, example is served only for explaining the present invention, and
It is non-for limiting the scope of the present invention.
As shown in Figure 1, a kind of internal storage data processing system, the system includes what is be connected with data creation module respectively
Communication module, data operation modules and data read module：
Communication module, for receiving various information and output result；
Data creation module, the data message for being received according to communication module determines the storage catalogue of data；By number
According to the form storage according to data file to storage server, temporal information is included in the filename of the data file；Institute
The storage server stated includes the disk array being made up of multiple disc drivers, carries out logic point to the disk array in advance
Area；
Data operation modules, for needing to data creation module in the operation in when adding token record, inquiry is described
One logic area of description information selection size matching, and will mark to record the memory block of free time in the logic area for writing selection
It is interior；
Data read module, for the multiple threads that read of the information creating in data creation module come successively from list
Middle extraction data file, reads the data in data creation module.
After logical partition is carried out, multistage file directory is set up by setting rule in each logical partition；
The division rule of data file is set up, makes the size of the data file of storage in the range of setting；
During data storage, before storage server data storage, first analyze data is simultaneously true according to the foundation rule of multistage catalogue
Determine storage catalogue；
It is to set up index by the data storage file of setting time；
Storage server is inquired about data and is built come the path of scanning index and data file using segmentation multi-thread concurrent technology
Lithol is cited as setting up multiple index.
Also include step：After index is set up, timing updates index.
Data creation module also includes：Determining unit, the size for determining data file in data creation module；
Creating unit, for setting up the first memory block, the second memory block and the 3rd memory block in shared drive；Storage unit, is used for
The description information of database is saved in the first memory block, the description information of database includes the description information of each database table.
Data operation modules also include：Operating unit, the token record for will need addition is write in the logic area of selection
In idle memory block, and the position letter that token record is preserved in the second memory block.
Memory database realizes the loose coupling of main memory database table structure and data store organisation, in storage table record
When, by choosing the memory module that the logical block matched with table record size can realize the variable-length record of same table, drop
The occupancy of table space resource in low shared drive；The mode of Dynamic expansion table space ensure that memory database in the present embodiment
Do not exist coupled relation between table structure and data store organisation, also, pass through a memory database process dynamic in system
When expanding table space, other memory database application processes update should in the timestamp of logical block and system definition in this process
The timestamp of logic will is consistent, it is ensured that the synchronization of memory headroom mapping.
Memory database disclosure satisfy that the requirement of real-time that data are carried out with a large amount of access, can be applicable to requirement of real time higher
System in, for example, the memory database that is provided using the present embodiment in a Real-time Billing System improves the reality of system
When response performance, the relevant information of database stores user and the charge information using business.
In this manual, the schematic representation to above-mentioned term is necessarily directed to identical embodiment or example.
And, the specific features of description, structure, material or feature can be in any one or more embodiments or example with suitable
Mode is combined.Additionally, in the case of not conflicting, those skilled in the art can be by the difference described in this specification
The feature of embodiment or example and different embodiments or example is combined and combines.
The foregoing is only presently preferred embodiments of the present invention, be not intended to limit the invention, it is all it is of the invention spirit and
Within principle, any modification, equivalent substitution and improvements made etc. should be included within the scope of the present invention.
Claims (6)
1. a kind of internal storage data processing system, it is characterised in that the system includes the communication being connected with data creation module respectively
Module, data operation modules and data read module：
Communication module, for receiving various information and output result；
Data creation module, the data message for being received according to communication module determines the storage catalogue of data；By data according to
The form of data file is stored in storage server, and temporal information is included in the filename of the data file；Described
Storage server includes the disk array being made up of multiple disc drivers, carries out logical partition to the disk array in advance；
Data operation modules, for needing to data creation module in the operation in when adding token record, inquire about the description
One logic area of information selection size matching, and token is recorded in the memory block of free time in the logic area for writing selection；
Data read module, reads thread to be carried from list successively for the information creating in data creation module to be multiple
Data file is taken, the data in data creation module are read.
2. a kind of internal storage data processing system according to claim 1, it is characterised in that for data creation module,
After carrying out logical partition, multistage file directory is set up by setting rule in each logical partition；
The division rule of data file is set up, makes the size of the data file of storage in the range of setting；
During data storage, before storage server data storage, first analyze data simultaneously determines to deposit according to the foundation rule of multistage catalogue
Storage catalogue；
It is to set up index by the data storage file of setting time；
Storage server inquires about data using segmentation multi-thread concurrent technology come scanning index and the path of data file.
3. a kind of internal storage data processing system according to claim 1, it is characterised in that described data creation module is also
Including：
Determining unit, the size for determining data file in data creation module；
Creating unit, for setting up the first memory block, the second memory block and the 3rd memory block in shared drive；
Storage unit, for the description information of database to be saved in into the first memory block, the description information of database includes each number
According to the description information of storehouse table.
4. a kind of internal storage data processing system according to claim 1, it is characterised in that described data operation modules are also
Including：
Operating unit, the token record for that will need addition is write in idle memory block in the logic area of selection, and the
The position letter of token record is preserved in two memory blocks.
5. a kind of internal storage data processing system according to claim 2, it is characterised in that for data creation module, institute
Foundation index is stated to set up multiple index.
6. a kind of internal storage data processing system according to claim 2, it is characterised in that for data creation module, rope
After drawing foundation, timing updates index.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611051186.9A CN106844418A (en) | 2016-11-25 | 2016-11-25 | A kind of internal storage data processing system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611051186.9A CN106844418A (en) | 2016-11-25 | 2016-11-25 | A kind of internal storage data processing system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106844418A true CN106844418A (en) | 2017-06-13 |
Family
ID=59145887
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611051186.9A Withdrawn CN106844418A (en) | 2016-11-25 | 2016-11-25 | A kind of internal storage data processing system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106844418A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109446179A (en) * | 2018-09-30 | 2019-03-08 | 天津天地伟业电子工业制造有限公司 | A method of it is concurrent that SQLITE database realizing multi-service being established based on Memory mode |
-
2016
- 2016-11-25 CN CN201611051186.9A patent/CN106844418A/en not_active Withdrawn
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109446179A (en) * | 2018-09-30 | 2019-03-08 | 天津天地伟业电子工业制造有限公司 | A method of it is concurrent that SQLITE database realizing multi-service being established based on Memory mode |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102129458B (en) | Method and device for storing relational database | |
| US9933979B2 (en) | Device and method for storing data in distributed storage system | |
| CN103294710B (en) | A kind of data access method and device | |
| CN104252536B (en) | A kind of internet log data query method and device based on hbase | |
| US8099421B2 (en) | File system, and method for storing and searching for file by the same | |
| CN102682108B (en) | Row and line mixed database storage method | |
| CN103838830A (en) | Data management method and system of HBase database | |
| CN105117417A (en) | Read-optimized memory database Trie tree index method | |
| CN100424699C (en) | Attribute extensible object file system | |
| CN102622434A (en) | Data storage method, data searching method and device | |
| CN107391544B (en) | Processing method, device and equipment of column type storage data and computer storage medium | |
| CN103838853A (en) | Mixed file system based on different storage media | |
| CN105677904A (en) | Distributed file system based small file storage method and device | |
| CN106909595A (en) | A kind of data migration method and device | |
| CN102902709A (en) | Space allocation fixing file memory system and implementation method | |
| CN107133183B (en) | Cache data access method and system based on TCMU virtual block device | |
| CN106951182A (en) | A kind of block device caching method and device | |
| CN104536908A (en) | Single-machine-oriented mass small record efficient storage and management method | |
| JP2012168781A (en) | Distributed data-store system, and record management method in distributed data-store system | |
| CN106844418A (en) | A kind of internal storage data processing system | |
| CN117608856A (en) | Memory expansion method, system, terminal and storage medium for NVMe (network video Me) acceleration card | |
| CN104462080A (en) | Index structure creating method and system with group statistics for search results | |
| US7627719B2 (en) | Cache device and method for determining LRU identifier by pointer values | |
| US7792798B2 (en) | Dynamic data compaction for data redistribution | |
| US20130031048A1 (en) | Data partitioning apparatus and data partitioning method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20170613 |