CN109240599A - A kind of big data stocking system is integrated - Google Patents
A kind of big data stocking system is integrated Download PDFInfo
- Publication number
- CN109240599A CN109240599A CN201810769163.4A CN201810769163A CN109240599A CN 109240599 A CN109240599 A CN 109240599A CN 201810769163 A CN201810769163 A CN 201810769163A CN 109240599 A CN109240599 A CN 109240599A
- Authority
- CN
- China
- Prior art keywords
- module
- data
- metadata
- library
- read
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0656—Data buffering arrangements
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
It is integrated that the invention discloses a kind of big data stocking systems, unit, encryption unit and server unit are logged in including user, it includes password recording module and account recording module that user, which logs in unit, encryption unit includes password authentication module, account verification module, error logging module, identifying code module and identifying code detection unit, and server unit includes account data library module, code data library module, IP acquiring unit, server logging modle, identifying code transmission unit, account locking module, backstage statistical module and resource consolidation module.A kind of big data stocking system of the present invention is integrated, equipped with server logging modle, IP acquiring unit, account locking module, backstage statistical module and resource consolidation module, statistic record can be carried out to abnormal account, abnormal account is verified by binding account by backstage, and has the function of locking account, bring better prospect of the application.
Description
Technical field
The present invention relates to field of data storage systems, in particular to a kind of big data stocking system is integrated.
Background technique
The data volume generated with the development of the technologies such as cloud computing, Internet of Things, mobile Internet, human society every field
All sharp increases, with the appearance of various emerging network services, the content of service is changed into respectively from single text formatting
Kind multimedia form, such as video, image, sound；Various users generate data, such as the novel social matchmaker of blog, microblogging, wechat
Development in body both increases user network behavioral data；Being widely used for internet of things sensors network has also automatically generated greatly
The data of amount.Explosive with big data application increases, it has derived oneself unique framework, and directly pushes away
The development of storage, network and computing technique is moved.Processing this special demand of big data is a new challenge after all.Firmly
The development of part is final or by software requirement promotion, and nowadays, big data has become a kind of important basic resource, and shape
At development trend data-centered, centered on storing, thus how to store data that these quantity rise sharply at
For the research hotspot of enterprise and academia, data storage is key link indispensable during big data is handled and utilized,
Although existing big data storage system can satisfy big data application memory demand, but its input/output (I/O) performance obtains
Less than optimization, directly determine that big data storage efficiency is lower, and influences big data application overall performance.
Summary of the invention
It is integrated that the main purpose of the present invention is to provide a kind of big data stocking systems, can effectively solve in background technique
The problem of.
To achieve the above object, the technical scheme adopted by the invention is as follows:
A kind of big data stocking system is integrated, including data center's library module, data transmission module and center management module, described
Data center's library module includes picture classification module, visual classification module and online backup module, the central data bank mould
The information that block is passed over by the data transmission module is by classification storage in the picture classification module, the video point
Generic module and the online backup inside modules, data center's library module are connect with data using objects, and the data make
The central data bank inside modules data can be read out or are written with object, the data using objects include
People user and enterprise customer, the data transmission module include framework optimization module, metadata I/O optimization module and data I/O
Optimization module, metadata and data input by the data transmission module or export data center's library module, described
Center management module includes metadata management module, system management module, Subject Manager module and data display module, it is described in
Heart management module is managed and shows to the data information of the central data bank inside modules.
Preferably, the metadata I/O optimization module includes metadata search module, metadata lookup module and metadata
Creation module, three respectively optimize the creation of metadata, improve the message data rate of metadata I/O optimization module,
The data transmission module is optimized the performance of metadata record heart library module in the database.
Preferably, the data I/O optimization module includes file-level slitting module, bandwidth allocation module, buffer memory capacity expansion
Open up module and read-write optimization module.
Preferably, the file-level slitting module, bandwidth allocation module, buffer memory capacity expansion module and read-write optimization module
When the data to database center library module are written and read, the bandwidth of the idle read-write of bandwidth allocation module distribution, file fraction
Data are carried out slitting by module, and read-write optimization module optimizes read-write, and buffer memory capacity expansion module expands when caching inadequate
The temporary capacity of big data I/O optimization module.
Preferably, the metadata management module, which is mainly responsible for, carries out pipe to the metadata of database consolidated storage inside modules
Reason, and the transmission rate of metadata I/O optimization module and metadata management module matching management data transmission module.
Preferably, the personal user connect with data display module, the enterprise customer respectively with metadata management mould
Block, system management module are connected with Subject Manager module, the data of personal user's read-only data library center library module, enterprise customer
The data of database hub library module can be read and write.
Compared with prior art, the invention has the following beneficial effects:
In the present invention, data transmission module is divided into framework optimization module, metadata I/O optimization module and data I/O and optimizes mould
Block constructs the general frame of data transmission, and metadata and data optimize when inputting, and metasearch mould is respectively set
Block, metadata lookup module and metadata creation module accelerate the lookup of metadata, mitigate first number inside center management module
According to the pressure of management module, improve the management performance of center management module, and file-level slitting module, bandwidth allocation module,
Buffer memory capacity expansion module and read-write optimization module integrally optimize data I/O optimization module, and writing for data is separately optimized
Enter the buffer memory capacity of rate, bandwidth occupancy than inputting with growth data when and improve data entry rate indirectly, finally again by data
Framework when transmission optimizes, and guarantees not lose when data transmission, enhances the performance of improve data transfer to improve big data
The overall performance of storage.
Detailed description of the invention
Fig. 1 is a kind of integrated overall structure diagram of big data stocking system of the present invention；
Fig. 2 is a kind of integrated center management module structure chart of big data stocking system of the present invention；
Fig. 3 is a kind of big data stocking system integrated data I/O optimization module structure chart of the present invention；
In figure: 1, data center's library module, 2, data transmission module, 3, center management module, 4, data using objects, 5, individual
User, 6, enterprise customer, 7, picture classification module, 8, visual classification module, 9, online backup module, 10, framework optimization module,
11, metadata I/O optimization module, 12, data I/O optimization module, 13, metadata search module, 14 metadata lookup modules,
15, metadata creation module, 16, metadata management module, 17, system management module, 18, Subject Manager module, 19, data exhibition
Show module, 20, file-level slitting module, 21, bandwidth allocation module, 22, buffer memory capacity expansion module, 23, read-write optimization module.
Specific embodiment
To be easy to understand the technical means, the creative features, the aims and the efficiencies achieved by the present invention, below with reference to
Specific embodiment, the present invention is further explained.
As shown in Figure 1-3, a kind of big data stocking system is integrated, including data center's library module 1, data transmission module 2
With center management module 3, data center's library module 1 includes picture classification module 7, visual classification module 8 and online backup mould
Block 9, the information that data center's library module 1 is passed over by data transmission module 2 is by classification storage in picture classification module
7, inside visual classification module 8 and online backup module 9, data center's library module 1 is connect with data using objects 4, and data make
1 internal data of data center's library module can be read out or is written with object 4, data using objects 4 include personal use
Family 5 and enterprise customer 6, data transmission module 2 include framework optimization module 10, metadata I/O optimization module 11 and data I/O excellent
Change module 12, metadata and data by the input of data transmission module 2 or output data center library module 1, centre management mould
Block 3 includes metadata management module 16, system management module 17, Subject Manager module 18 and data display module 19, central tube
Reason module 3 is managed and shows to the data information inside central data bank module 1.
Metadata I/O optimization module 11 includes metadata search module 13, metadata lookup module 14 and metadata creation
Module 15, three respectively optimize the creation of metadata, improve the message data rate of metadata I/O optimization module 11,
Data transmission module 2 is optimized the performance of metadata record heart library module 1 in the database, data I/O optimization module
12 include file-level slitting module 20, bandwidth allocation module 21, buffer memory capacity expansion module 22 and read-write optimization module 23, file
Grade slitting module 20, bandwidth allocation module 21, buffer memory capacity expansion module 22 and read-write optimization module 23 are to database hub
When the data of library module 1 are written and read, bandwidth allocation module 21 distributes the bandwidth of idle read-write, and file-level slitting module 20 will count
According to slitting is carried out, read-write 23 pairs of read-writes of optimization module are optimized, and buffer memory capacity expansion module 22 expands data when caching inadequate
The temporary capacity of I/O optimization module 12, metadata management module 16 are mainly responsible for first number inside database center library module 1
According to being managed, and the transmission of metadata I/O optimization module 11 and 16 matching management data transmission module 2 of metadata management module
Rate, personal user 6 connect with data display module 19, enterprise customer 6 respectively with metadata management module 16, system administration mould
Block 17 and Subject Manager module 18 connect, and the data of 6 read-only data library center library module 1 of personal user, enterprise customer 6 can read
Write the data of database hub library module 1.
It should be noted that the present invention is integrated for a kind of big data stocking system, when in use, data using objects 4 can
To be written and read by data transmission module 2 to the data inside central data bank module 1, wherein enterprise customer 6 can pass through
Metadata management module 16, system management module 17 and Subject Manager module 18 inside central management system 3 is to data center
Data in library module 1 are written and read, and personal user 5 can pass through the data display module in central management system 3
19 read the data in data center's library module 1, and personal user 5 is avoided to modify the data of data center's library module 1 privately, and read
The data write carry out classified storage by center management module 3 and online backup module 9, and picture puts into picture classification module 7, depending on
In frequency investment visual classification module 8, data transmission module 2 passes through framework optimization module 10,11 sum number of metadata I/O optimization module
It is optimized according to I/O optimization module 12, the efficiency of improve data transfer, the overall performance of enhancing big data storage.
The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention.The technology of the industry
Personnel are it should be appreciated that the present invention is not limited to the above embodiments, and the above embodiments and description only describe this
The principle of invention, without departing from the spirit and scope of the present invention, various changes and improvements may be made to the invention, these changes
Change and improvement all fall within the protetion scope of the claimed invention.The claimed scope of the invention by appended claims and its
Equivalent thereof.
Claims (6)
1. a kind of big data stocking system is integrated, including data center's library module (1), data transmission module (2) and centre management
Module (3), it is characterised in that: data center's library module (1) includes picture classification module (7), visual classification module (8)
The information passed over online backup module (9), data center's library module (1) by the data transmission module (2)
By classification storage in the picture classification module (7), the visual classification module (8) and the online backup module (9)
Portion, data center's library module (1) connect with data using objects (4), and the data using objects (4) can be by the number
Be read out or be written according to center library module (1) internal data, the data using objects (4) include personal user (5) and
Enterprise customer (6), the data transmission module (2) include framework optimization module (10), metadata I/O optimization module (11) sum number
According to I/O optimization module (12), metadata and data input or export the data center by the data transmission module (2)
Library module (1), the center management module (3) include metadata management module (16), system management module (17), Subject Manager
Module (18) and data display module (19), the center management module (3) number internal to data center's library module (1)
It is believed that breath is managed and shows.
2. a kind of big data stocking system according to claim 1 is integrated, it is characterised in that: the metadata I/O optimization
Module (11) includes metadata search module (13), metadata lookup module (14) and metadata creation module (15), San Zhefen
The other creation to metadata optimizes, and improves the message data rate of metadata I/O optimization module (11), passes to the data
Defeated module (2) optimizes the performance of metadata record heart library module (1) in the database.
3. a kind of big data stocking system according to claim 1 is integrated, it is characterised in that: the data I/O optimizes mould
Block (12) includes file-level slitting module (20), bandwidth allocation module (21), buffer memory capacity expansion module (22) and read-write optimization
Module (23).
4. a kind of big data stocking system according to claim 1 or 3 is integrated, it is characterised in that: the file-level slitting
Module (20), bandwidth allocation module (21), buffer memory capacity expansion module (22) and read-write optimization module (23) are in database
When the data of heart library module (1) are written and read, the bandwidth of the idle read-write of bandwidth allocation module (21) distribution, file-level slitting module
(20) data are subjected to slitting, read-write optimization module (23) optimizes read-write, buffer memory capacity expansion module when caching inadequate
(22) expand the temporary capacity of data I/O optimization module (12).
5. a kind of big data stocking system according to claim 1 is integrated, it is characterised in that: the metadata management module
(16) it is mainly responsible for the metadata internal to database center library module (1) to be managed, and metadata I/O optimization module (11)
With the transmission rate of metadata management module (16) matching management data transmission module (2).
6. a kind of big data stocking system according to claim 1 is integrated, it is characterised in that: the personal user (6) with
Data display module (19) connection, the enterprise customer (6) respectively with metadata management module (16), system management module (17)
It is connected with Subject Manager module (18), the data of personal user (6) read-only data library center library module (1), enterprise customer (6) can
To read and write the data of database hub library module (1).
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810769163.4A CN109240599A (en) | 2018-07-13 | 2018-07-13 | A kind of big data stocking system is integrated |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810769163.4A CN109240599A (en) | 2018-07-13 | 2018-07-13 | A kind of big data stocking system is integrated |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109240599A true CN109240599A (en) | 2019-01-18 |
Family
ID=65072614
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810769163.4A Pending CN109240599A (en) | 2018-07-13 | 2018-07-13 | A kind of big data stocking system is integrated |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109240599A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110069501A (en) * | 2019-04-23 | 2019-07-30 | 北京恒冠网络数据处理有限公司 | A kind of big data stocking system |
| CN112053038A (en) * | 2020-08-18 | 2020-12-08 | 广东电网有限责任公司广州供电局 | Migration and modification system with big data integration display function |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102307236A (en) * | 2011-08-25 | 2012-01-04 | 浪潮电子信息产业股份有限公司 | Cloud-storage-structure-based remote storage management system |
| CN103677673A (en) * | 2013-12-23 | 2014-03-26 | Tcl集团股份有限公司 | Method and system for uploading files in classifying and batching mode |
| US20150363115A1 (en) * | 2014-06-12 | 2015-12-17 | Lenovo Enterprise Solutions (Singapore) Pte. Ltd | Parallel storage system testing |
| CN105302920A (en) * | 2015-11-23 | 2016-02-03 | 上海爱数信息技术股份有限公司 | Optimal management method and system for cloud storage data |
| CN107609166A (en) * | 2017-09-27 | 2018-01-19 | 合肥博力生产力促进中心有限公司 | A kind of data dispatch system based on cloud management platform |
| CN107797768A (en) * | 2017-10-11 | 2018-03-13 | 南京东方金信数据服务有限公司 | A kind of method and system for handling big data |
-
2018
- 2018-07-13 CN CN201810769163.4A patent/CN109240599A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102307236A (en) * | 2011-08-25 | 2012-01-04 | 浪潮电子信息产业股份有限公司 | Cloud-storage-structure-based remote storage management system |
| CN103677673A (en) * | 2013-12-23 | 2014-03-26 | Tcl集团股份有限公司 | Method and system for uploading files in classifying and batching mode |
| US20150363115A1 (en) * | 2014-06-12 | 2015-12-17 | Lenovo Enterprise Solutions (Singapore) Pte. Ltd | Parallel storage system testing |
| CN105302920A (en) * | 2015-11-23 | 2016-02-03 | 上海爱数信息技术股份有限公司 | Optimal management method and system for cloud storage data |
| CN107609166A (en) * | 2017-09-27 | 2018-01-19 | 合肥博力生产力促进中心有限公司 | A kind of data dispatch system based on cloud management platform |
| CN107797768A (en) * | 2017-10-11 | 2018-03-13 | 南京东方金信数据服务有限公司 | A kind of method and system for handling big data |
Non-Patent Citations (1)
| Title |
|---|
| 肖利民等: ""大数据存储系统I_O性能优化技术研究进展"", 《大数据》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110069501A (en) * | 2019-04-23 | 2019-07-30 | 北京恒冠网络数据处理有限公司 | A kind of big data stocking system |
| CN112053038A (en) * | 2020-08-18 | 2020-12-08 | 广东电网有限责任公司广州供电局 | Migration and modification system with big data integration display function |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107465656B (en) | Security monitoring big data processing method and system based on cloud computing | |
| CN105357201B (en) | A kind of object cloud storage access control method and system | |
| CN106776780B (en) | Data exchange and sharing method and system in cloud environment | |
| CN104252536B (en) | A kind of internet log data query method and device based on hbase | |
| CN103312791B (en) | Internet of Things isomeric data storage means and system | |
| CN106951552A (en) | A kind of user behavior data processing method based on Hadoop | |
| CN107315776A (en) | A kind of data management system based on cloud computing | |
| CN109947668A (en) | The method and apparatus of storing data | |
| WO2014015488A1 (en) | Method and apparatus for data storage and query | |
| CN113220794A (en) | Method, server and computer readable medium for synchronizing query results between heterogeneous databases | |
| CN105740337A (en) | Rapid event matching method in content-based publishing subscription system | |
| CN104778229A (en) | Telecommunication service small file storage system and method based on Hadoop | |
| CN104142930A (en) | Universal Delta data loading technology | |
| CN107832392A (en) | A kind of metadata management system | |
| CN105354250A (en) | Data storage method and device for cloud storage | |
| CN106547911A (en) | A kind of access method and system of mass small documents | |
| CN104615785A (en) | Data storage method and device based on TYKY cNosql | |
| CN115859362A (en) | Data storage system, method, device and medium based on block chain side chain | |
| CN109636605A (en) | A kind of monitoring and managing method and system of digital asset exchange | |
| WO2023185111A1 (en) | Quick access method and device for data file | |
| CN107273368A (en) | Mass data storage management method, device and data-storage system | |
| CN109240599A (en) | A kind of big data stocking system is integrated | |
| CN110377757A (en) | A kind of real time knowledge map construction system | |
| CN106875275A (en) | The on-line automatic reconciliation system and method for commercial health insurance | |
| CN109947743A (en) | A kind of the NoSQL big data storage method and system of optimization |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190118 |