CN108228645A - Big data document library system - Google Patents
Big data document library system Download PDFInfo
- Publication number
- CN108228645A CN108228645A CN201611193201.3A CN201611193201A CN108228645A CN 108228645 A CN108228645 A CN 108228645A CN 201611193201 A CN201611193201 A CN 201611193201A CN 108228645 A CN108228645 A CN 108228645A
- Authority
- CN
- China
- Prior art keywords
- data
- request
- host
- adapter coupling
- online
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of big data document library systems, it is established on the Language Infrastructure of SQL database and NoSQL databases, for carrying out the access of data after multiple client log-in, it is equipped with an at least adapter coupling servers and a data host, multiple queue devices and an assignor are equipped in the adapter coupling servers, multiple queue device connects the corresponding client respectively, the assignor connects multiple queue device and the data host, when multiple client is respectively transmitted a request of data, corresponding queue device is received request of data and is arranged multiple request of data in the form of queue, and the assignor controls multiple request of data online to be sent to the data host by single between the adapter coupling servers and the data host respectively.In this way, the online load of the data host of terminal can be reduced, so as to promote the working efficiency of total system.
Description
Technical field
The present invention relates to this technical field of database, especially with regard to a kind of big data document library system, to establish
Relational database (the Relational of SQL (Structural Query Language, structured query language)
Database) with non-relational database (NoSQL database) technically, arranged using an intermediate device and assign online task,
So as to mitigate the online load of end host, and then promote the working efficiency in overall data library.
Background technology
2012,《The New York Times》A column《The Age of Big Data》Intimation big data
The arriving in epoch, so far, " big data " word has been that what's frequently heard can be repeated in detail, and at present, the data volume accumulated in the world is more and more huger,
Database also faces the challenge of high read-write demand, high storage efficiency, high requirements for access and high scalability.With general relation data
For library, synchronous and symmetrical online framework is used, so as to be carried out by the online number of corresponding client (Client) quantity
Data access, to realize powerful data query ability.However, such distributed data base for belonging to single master-slave relationship is because of it
Online number need to be with client equivalent, therefore is limited by hardware device load-bearing capacity, in facing the larger data access demand of quantity
When, it certainly will need that original hardware device is expanded to or replaced the hardware device of more high standard, thus, in addition to cost
Outside increasing, also need to face the difficulty in operation.
In addition, though non-relational database is using distributed storage and the operating type of operation processing data and with computer hardware
Demand it is cheap with high-level expansion capacity the characteristics of, it is however, similary using synchronous and symmetrical online framework, therefore in facing number
Still need when measuring larger data access demand and expand original hardware device, and since non-relational database uses
New program language, therefore in addition to query capability is extremely weak, high error rate, and its language can be led to when the write-in output of mass data
Maturity is insufficient and the risk in edition upgrading easily occurs, and also reduces safety and the stability of database, is unfavorable for high wind
The industry development of dangerous demand.
For this purpose, in view of the real-time of relational database, accuracy and safety and the high extendibility of non-relational database
With high availability, that is, SQL+NoSQL databases 1 as shown in Figure 1 are developed, multiple client 10 is online to a SQL respectively
Database 11, which is online to a NoSQL databases 12 again, and the NoSQL databases 12 are stored with a tables of data
(not shown), for data query.Since multiple client 10 is linked by sql like language and the SQL database 11, SQL numbers
Translation communication is carried out, therefore data query ability is still on the weak side with new program language with the NoSQL databases 12 again according to library 11.Separately
Outside, which is still synchronous and symmetrical online framework, though the ability with horizontal expandable, is limited to this
Communication between SQL database 11 and the NoSQL databases 12 need to pass through translation, therefore for total system, the data after expansion
Higher work load is born in library instead, and query time is caused to extend, so as to be unfavorable for industry application.
Thoughts on this, how to improve existing database schema or propose different frameworks, in needed for reduction database
Load it is online several while, it is ensured that data (Data) are converted into the working efficiency of information (Information), as of the invention
Desire most ardently the subject probed into.
Invention content
In view of the above problem, the purpose of the present invention is to provide a kind of high extendibility and big data document library system,
While the network segment load that data host is accepted is reduced, to make total system that can at least handle the number of the online number of 400K user
According to request, the high quality operation of mass data is realized.
Purpose according to the present invention, the big data document library system establish the language in SQL database and NoSQL databases
On framework, for carrying out the access of data after multiple client log-in, wherein：The big data document library system is equipped at least one
Adapter coupling servers and a data host, the adapter coupling servers are interior to be equipped with multiple queue devices and an assignor, multiple queue device point
The corresponding client is not connected, which connects multiple queue device and the data host, when multiple client is distinguished
When transmitting a request of data, corresponding queue device is received request of data and is arranged multiple request of data in the form of queue
Row, and the assignor controls multiple request of data to pass through the single online biography between the adapter coupling servers and the data host respectively
It send to the data host.
Wherein, which reads suitable after multiple request of data according to the mechanism of first in first out from multiple queue device
Sequence is sent to the data host, and multiple queue device corresponds to multiple client, each online corresponding queue device of client, and works as
When the adapter coupling servers are α, the online number of the network segment between the data host and multiple adapter coupling servers is also α, and wherein α is big
In 1 integer.
Also, using relational database architecture between the adapter coupling servers and multiple client, the adapter coupling servers are with being somebody's turn to do
Non-relational database framework is used between data host.
In conclusion the present invention carries out the translation work of new program language by the assignor, it is existing so as to solve
The problem of NoSQL databases low query function, and it is multiple by multiple queue device and the use sequential processes of the assignor
Request of data, so as to solve the connection of in the past SQL database, NoSQL databases or SQL+NoSQL database demand peer users numbers
Machine amount problem, also that is, the processing and response of the corresponding thread progress data of respectively request of data distribution need not be given, in this way, i.e.
Make multiple client quantity increase or the quick-fried increasing of multiple request of data, the data host still under the coordination of the assignor,
By the processing and response of the single online carry out data between the adapter coupling servers, from loading without the network segment is online
The problem of high.Further, big data document library system provided by the invention can be used for information exchange and at various data
The collocation of reason method is implemented.
Description of the drawings
Fig. 1 is the configuration diagram of existing SQL+NoSQL databases；
Fig. 2 is the configuration diagram of present pre-ferred embodiments；
Fig. 3 is the flow diagram of present pre-ferred embodiments；
Fig. 4 uses schematic diagram for present pre-ferred embodiments；
Fig. 5 is the application schematic diagram of present pre-ferred embodiments.
Reference sign：1-SQL+NoSQL databases；10- clients；11-SQL databases；12-NoSQL databases；
2- big data document library systems；20- clients；21- adapter coupling servers；210- queue devices；211- assignors；22- data hosts；
3- Cloud Servers；S1~S3- steps.
Specific embodiment
For auditor is enable to have a clear understanding of present disclosure, schema of sincerely arranging in pairs or groups in the following instructions please refer to.
It is respectively the configuration diagram and flow diagram of present pre-ferred embodiments as shown in Figure 2 and Figure 3.As shown in the figure,
The big data document library system 2 is established on the Language Infrastructure of SQL database and NoSQL databases, please for multiple client 20
The access of data is carried out after asking online, is equipped with an at least adapter coupling servers 21 and a data host 22.The adapter coupling servers 21
Interior to be equipped with multiple 210 and one assignors 211 of queue device, multiple queue device 210 connects the corresponding client 20 and leads to respectively
The language technology progress structure for crossing relational database architecture leads to, which connects multiple queue device 210 and the data host
22, and pass through non-relational database framework language technology and the data host 22 carry out structure lead to, and the big data data bank system
It may include the following steps during 2 work of system.
When the quantity of multiple client 20 is N, in step S1, the setting multiple teams identical with 20 quantity of client
Row device 210 makes have N number of queue device 210 in the adapter coupling servers 21, and respectively online corresponding respectively client of the queue device 210
End 20, also that is, to present N items online for the network segment between the adapter coupling servers 21 and multiple client 20.
Then, when multiple client 20 is respectively transmitted a request of data, step S2, corresponding queue device 210 connects
It receives request of data and arranges multiple request of data in the form of queue, also that is, the instruction from the respectively client 20, friendship
Easily or request is used in sql like language write-in respectively the queue device 210.Meanwhile respectively one request of the queue device 210 transmission notifies to this to refer to
Device 211 is sent, in multiple request of data of flood tide and while be stored in the adapter coupling servers 21, create the assignor 211
A thread of multiple request of data is performed stage by stage.
In step S3, which reads multiple according to the mechanism of first in first out from multiple queue device 210
Request of data, and after being translated into the program language that the data host 22 can recognize, pass through the adapter coupling servers 21 and the data
Single online sequence between host 22 is distributed to the data host 22, in this way, corresponding to the respectively queue device 210 by this mechanism coordination
The work put successively of cross-thread difference, you can the data host 22 is allowed to have when flood tide request is faced still can be with single
The function that machine is received and replied influences so as to which the request of data that multiple client 20 inputs will not be caused sequential overlapping occur
The problem of system operations timeliness, occurs.Incidentally, by the use of the assignor 211 as the adapter coupling servers 21 and the data
Intermediate device between host 22 carries out the data processing of the translation of program language, and the big data document library system 2 can be substantially improved
Query capability, and the online number between multiple client 20 and multiple adapter coupling servers 21 is sufficiently different from multiple turn
The online number between server 21 and the data host 22 is connect, therefore the network segment load that the data host 22 is accepted can be greatly reduced.
Also, when the quantity of multiple client 20 reaches a degree and the adapter coupling servers 21 is made to reach the online upper limit
When, which can augment time adapter coupling servers 21 again, and secondary one adapter coupling servers 21 receive automatically
It, should also through its single online sequence distribution between the data host 22 after the request of data of multiple client 20 of excess
Multiple request of data give the data host 22.As shown in figure 4, if respectively the assignor 210 is all merely responsible for 4,000, i.e. the online numbers of 4K,
Such as when the adapter coupling servers 21 such as 3 make that multiple assignor 210 is total 100, though the data host 22 with
The online number of the network segment between multiple adapter coupling servers 21 is 3 online numbers, which can still handle 400K connection
Machine number, greatly in the online several grades of 10K of NoSQL databases now, can unquestionably be solved without mistake and problem
The instruction of flood tide and data problem.In other words, when multiple adapter coupling servers 21 are α, the data host 22 and multiple turn
The online number of the network segment connect between server 21 is also α, forms asynchronous and asymmetric online system architecture.Divide since the present invention uses
Cloth calculation, and respectively the adapter coupling servers 21 only need to handle the sequence of multiple request of data and translation work, therefore respectively should
The hsrdware requirements of adapter coupling servers 21 are not high, can increase and decrease with the quantity of multiple client 20 and arbitrarily increase and decrease, can't shadow
Data operation and the storage of the rear end data host 22 are rung, and the data host 22 is primarily as data (Data after compression
Table storage), and respectively the adapter coupling servers 21 then all have read-write capability, unconventional master-slave system framework.
In addition, the big data document library system 2 should can be applied on cloud database as shown in Figure 5, in multiple turn
It connects 21 front end of server and disposes a Cloud Server 3, for providing industry application (Industry Applications), data point
Analysis application (Data Analytics Applications), function application (Functional Applications), online point
The services such as (Content Analytics) and business intelligence report (BI/Reporting) are analysed, so as to allow multiple client
20 by the connection mode of cloud network using the Cloud Server 3 enter the data host 22 carry out the inquiry of data, read or
Write-in etc..Multiple slot data (not shown) of corresponding multiple adapter coupling servers 21 can be arranged in the data host 22 again, it should
Data update synchronized with each other in multiple slot datas, for inquiring data in each self-corresponding slot data of the assignor 211, so as to
Promote working efficiency.
The foregoing is merely the preferred embodiment of illustrative rather than to be restricted.It is any without departing from the present invention spirit with
Scope, and the equivalent modifications carried out to it or change, are intended to be limited solely by this case scope of the claims.
Claims (6)
1. a kind of big data document library system is established on the Language Infrastructure of SQL database and NoSQL databases, confession is multiple
The access of data is carried out after client request is online, it is characterised in that：
The big data document library system is equipped with an at least adapter coupling servers and a data host, is equipped in the adapter coupling servers multiple
Queue device and an assignor, multiple queue device connect the corresponding client respectively, which connects multiple queue device
With the data host, when multiple client is respectively transmitted a request of data, corresponding queue device receives request of data simultaneously
Multiple request of data is arranged in the form of queue, and the assignor controls multiple request of data to pass through the switching respectively
Single between server and the data host online is sent to the data host.
2. big data document library system according to claim 1, which is characterized in that machine of the assignor according to first in first out
System is sequentially sent to the data host after multiple request of data is read from multiple queue device.
3. big data document library system according to claim 2, which is characterized in that multiple queue device corresponds to multiple visitor
Family end, each online corresponding queue device of client.
4. big data document library system according to claim 3, which is characterized in that, should when the adapter coupling servers are α
The online number of the network segment between data host and multiple adapter coupling servers is also α, and wherein α is the integer more than 1.
5. big data document library system according to claim 4, which is characterized in that the adapter coupling servers and multiple client
Relational database architecture is used between end.
6. big data document library system according to claim 5, which is characterized in that the adapter coupling servers and the data host
Between use non-relational database framework.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611193201.3A CN108228645A (en) | 2016-12-21 | 2016-12-21 | Big data document library system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611193201.3A CN108228645A (en) | 2016-12-21 | 2016-12-21 | Big data document library system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108228645A true CN108228645A (en) | 2018-06-29 |
Family
ID=62656705
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611193201.3A Pending CN108228645A (en) | 2016-12-21 | 2016-12-21 | Big data document library system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108228645A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6633914B1 (en) * | 1998-08-05 | 2003-10-14 | International Business Machines Corporation | Systems, methods and computer program products for handling client requests for server application processing using a thread pool |
| CN102308297A (en) * | 2011-07-13 | 2012-01-04 | 华为技术有限公司 | Data migration method, data migration device and data migration system |
| CN102323945A (en) * | 2011-09-02 | 2012-01-18 | 南京中兴力维软件有限公司 | SQL (Structured Query Language)-based database management method and device |
| US20130110853A1 (en) * | 2011-10-31 | 2013-05-02 | Microsoft Corporation | Sql constructs ported to non-sql domains |
| CN105468619A (en) * | 2014-09-03 | 2016-04-06 | 阿里巴巴集团控股有限公司 | Resource distribution method and device used for database connection pool |
-
2016
- 2016-12-21 CN CN201611193201.3A patent/CN108228645A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6633914B1 (en) * | 1998-08-05 | 2003-10-14 | International Business Machines Corporation | Systems, methods and computer program products for handling client requests for server application processing using a thread pool |
| CN102308297A (en) * | 2011-07-13 | 2012-01-04 | 华为技术有限公司 | Data migration method, data migration device and data migration system |
| CN102323945A (en) * | 2011-09-02 | 2012-01-18 | 南京中兴力维软件有限公司 | SQL (Structured Query Language)-based database management method and device |
| US20130110853A1 (en) * | 2011-10-31 | 2013-05-02 | Microsoft Corporation | Sql constructs ported to non-sql domains |
| CN105468619A (en) * | 2014-09-03 | 2016-04-06 | 阿里巴巴集团控股有限公司 | Resource distribution method and device used for database connection pool |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107506451B (en) | Abnormal information monitoring method and device for data interaction | |
| US9672272B2 (en) | Method, apparatus, and computer-readable medium for efficiently performing operations on distinct data values | |
| CN106503912A (en) | A kind of data service system | |
| CN107391653A (en) | A kind of distributed NewSQL Database Systems and image data storage method | |
| CN111639078A (en) | Data query method and device, electronic equipment and readable storage medium | |
| Gürcan et al. | Real-time processing of big data streams: Lifecycle, tools, tasks, and challenges | |
| CN113312191B (en) | Data analysis method, device, equipment and storage medium | |
| CN103262062A (en) | Systems and methods for performing a nested join operation | |
| US11954084B2 (en) | Method and apparatus for processing table, device, and storage medium | |
| CN112416991A (en) | Data processing method and device and storage medium | |
| US11442930B2 (en) | Method, apparatus, device and storage medium for data aggregation | |
| CN102868601B (en) | Routing system related to network topology based on graphic configuration database businesses | |
| CN104408183B (en) | The data lead-in method and device of data system | |
| CN106802865B (en) | Response simulator and method for software test | |
| US20220043773A1 (en) | Information processing method, electronic device, and storage medium | |
| CN208207819U (en) | A kind of big data analysis processing system based on extended node cluster | |
| CN109165263A (en) | Log analysis processing method and processing device, DDL synchronous method and the device of database | |
| CN105723365B (en) | Method for optimum indexing, main database node and subscriber database node | |
| CN117591516A (en) | Supervision report data analysis system, supervision report data analysis method, supervision report data analysis equipment and storage medium | |
| CN109542866A (en) | Vehicle location monitor cloud framework, method, server and medium | |
| CN106980617A (en) | A kind of method and system based on JSON sentence operating databases | |
| CN117851163A (en) | Service inspection tool based on self-grinding openstack platform | |
| CN108228645A (en) | Big data document library system | |
| EP4432117A1 (en) | Data model generation method and apparatus | |
| WO2021227636A1 (en) | Microservice processing method and apparatus, storage medium, and electronic device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20180629 |