CN106844496A - Data transmission scheduling method, device and server based on ESB - Google Patents
Data transmission scheduling method, device and server based on ESB Download PDFInfo
- Publication number
- CN106844496A CN106844496A CN201611217919.1A CN201611217919A CN106844496A CN 106844496 A CN106844496 A CN 106844496A CN 201611217919 A CN201611217919 A CN 201611217919A CN 106844496 A CN106844496 A CN 106844496A
- Authority
- CN
- China
- Prior art keywords
- data
- service
- operation request
- model
- data operation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of data transmission scheduling method based on ESB, device and server, including：Judge whether data corresponding with data operation request are structural data；If so, then calling first service；First service is used for when data corresponding with data operation request need to perform Hadoop operations, by Hive database response data operation requests；If it is not, second service is then called, by Hbase database response data operation requests.Interacting for structural data and Hadoop is realized in traditional forms of enterprises's service bus, solve the problems, such as that traditional database storage capacity is not enough, data analysis capabilities are not enough, improve the storage capacity and computational efficiency of data, it has been also convenient for the treatment of unstructured data, and it is packaged into service by interaction, facilitate the exploitation of Web Service, call, adapted to business rapid response to customer's need, improve the efficiency and scalability of system.
Description
Technical field
The present invention relates to technical field of data transmission, more specifically to a kind of data based on ESB
Transmission dispatching method, device and server.
Background technology
The product that ESB is traditional middleware technology to be combined with the technology such as XML, Web service.Enterprises service is total
Line, there is provided connection maincenter most basic in network, is the essential elements for constructing nervous system of enterprise.Functionally see, enterprise's clothes
Business bus provides the tupe that event-driven and document are oriented to, and distributed running fluidization air flow, and it supports to be based on
The route of content and filtering, the transmittability for possessing complex data, it is possible to which a series of standard interface is provided.In enterprise's clothes
Information content is sharply increased in business bus, increasing enterprise find existing conventional data storage, processing mode without
Method deals with the growth rate of data, cannot more meet demand of the user to data processing.
Therefore, how to solve the problems, such as that traditional database storage capacity and data analysis capabilities are not enough, improve system treatment
Efficiency, is that those skilled in the art need to solve.
The content of the invention
It is an object of the invention to provide a kind of data transmission scheduling method based on ESB, device and service
Device, to realize solving the problems, such as that traditional database storage capacity and data analysis capabilities are not enough, improves system treatment effeciency.
To achieve the above object, the embodiment of the invention provides following technical scheme：
A kind of data transmission scheduling method based on ESB, including：
Receive data operation request；
Judge whether data corresponding with the data operation request are structural data；
If so, then calling first service；The first service is used to be needed when data corresponding with the data operation request
When performing Hadoop operations, by data operation request described in Hive database responses；
If it is not, then calling second service；The second service passes through data operation request described in Hbase database responses.
Wherein, if data corresponding with the data operation request be structural data, and with the data operation request
When corresponding data need to carry out traditional operation, the data operation request is responded by traditional database.
Wherein, before the reception data operation request, also include：
The first model is configured in first device, first model is used for by Sqoop instruments, by traditional database
Data pick-up to Hive databases, response of the Hive databases to data operation request is realized by Hive scripts；
The first model in the first device is packaged into the first service, and the first service is deployed to clothes
Business device.
Wherein, before the reception data operation request, also include：
The second model is configured in second device, and second model is used to data are grasped by Hbase database realizings
The response that work is asked, completes the distributed operation of data；
The second model in the second device is packaged into the second service, and the second service is deployed to clothes
Business device.
A kind of data transmission scheduling device based on ESB, including：
Receiver module, for receiving data operation request；
Judge module, for judging whether data corresponding with the data operation request are structural data；
First scheduler module, during for data corresponding with the data operation request for structural data, calls first
Service；The first service is used to, when data corresponding with the data operation request need to perform Hadoop operations, pass through
Data operation request described in Hive database responses；
When second scheduler module for data corresponding with the data operation request is unstructured data, the is called
Two services；The second service passes through data operation request described in Hbase database responses.
Wherein, also include：
Performing module, for data corresponding with the data operation request be structural data, and with the data
When the corresponding data of operation requests need to carry out traditional operation, the data operation request is responded by traditional database.
Wherein, also include：
First model configuration module, for configuring the first model in first device, first model is used to pass through
Sqoop instruments, by the data pick-up in traditional database to Hive databases, Hive database logarithms are realized by Hive scripts
According to the response of operation requests；
First deployment module, for the first model in the first device to be packaged into the first service, and by institute
State first service and be deployed to server.
Wherein, also include：
Second model configuration module, for configuring the second model in second device, second model is used to pass through
Response of the Hbase database realizings to data operation request, completes the distributed operation of data；
Second deployment module, for the second model in the second device to be packaged into the second service, and by institute
State second service and be deployed to server.
A kind of server, including above-mentioned any one data transmission scheduling device.
By above scheme, a kind of data transmission scheduling based on ESB provided in an embodiment of the present invention
Method, including：Receive data operation request；Judge whether data corresponding with the data operation request are structural data；
If so, then calling first service；The first service is used to need to perform when data corresponding with the data operation request
When Hadoop is operated, by data operation request described in Hive database responses；If it is not, then calling second service；Described second
Service passes through data operation request described in Hbase database responses；
It can be seen that, in scheme, interacting for structural data and Hadoop is realized in traditional forms of enterprises's service bus, solve to pass
The problem that system database purchase scarce capacity, data analysis capabilities are not enough, improves the storage capacity and computational efficiency of data,
The treatment of unstructured data is facilitated, and service is packaged into by interaction, facilitate opening for Web Service
Send out, call, adapted to business rapid response to customer's need, improve the efficiency and scalability of system；The invention also discloses one kind
Data transmission scheduling device and server based on ESB.
Brief description of the drawings
In order to illustrate more clearly about the embodiment of the present invention or technical scheme of the prior art, below will be to embodiment or existing
The accompanying drawing to be used needed for having technology description is briefly described, it should be apparent that, drawings in the following description are only this
Some embodiments of invention, for those of ordinary skill in the art, on the premise of not paying creative work, can be with
Other accompanying drawings are obtained according to these accompanying drawings.
Fig. 1 is that a kind of data transmission scheduling method flow based on ESB is illustrated disclosed in the embodiment of the present invention
Figure；
Fig. 2 is that the disclosed scheduling mechanism operating process carried out data transmission based on Hadoop clusters of the embodiment of the present invention is shown
It is intended to；
Fig. 3 is a kind of date storage method schematic flow sheet disclosed in the embodiment of the present invention；
Fig. 4 is that a kind of data transmission scheduling apparatus structure based on ESB is illustrated disclosed in the embodiment of the present invention
Figure.
Specific embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme in the embodiment of the present invention is carried out clear, complete
Site preparation is described, it is clear that described embodiment is only a part of embodiment of the invention, rather than whole embodiments.It is based on
Embodiment in the present invention, it is every other that those of ordinary skill in the art are obtained under the premise of creative work is not made
Embodiment, belongs to the scope of protection of the invention.
The embodiment of the invention discloses a kind of data transmission scheduling method based on ESB, device and service
Device, to realize solving the problems, such as that traditional database storage capacity and data analysis capabilities are not enough, improves system treatment effeciency.
Referring to Fig. 1, a kind of data transmission scheduling method based on ESB provided in an embodiment of the present invention, bag
Include：
S101, reception data operation request；
S102, judge whether data corresponding with the data operation request are structural data；
If so, then performing S103；If it is not, then performing S104；
S103, call first service；The first service is used to be needed when data corresponding with the data operation request
When performing Hadoop operations, by data operation request described in Hive database responses；
Specifically, in scheme, Hive is built upon the data warehouse base frame on Hadoop, we can be Hive
Middle massive structured data regards table one by one as, and actually these data be distributed storage in HDFS, it is provided
Complete SQL query function, can be converted into Map/Reduce tasks by SQL statement.
S104, call second service；The second service passes through data operation request described in Hbase database responses.
Specifically, HBase is the sub-project of the Hadoop projects of Apache.HBase is different from general relational database,
It is a database for being suitable for unstructured data storage；HBase unlike another is per-column rather than being based on
Capable pattern.
It can be seen that, the tune that a kind of utilization Hadoop clusters in ESB carry out data transmission is disclosed in the present embodiment
Degree mechanism, realizes interacting for structural data and Hadoop in traditional forms of enterprises's service bus, and the quantitative change of reply data is big, treatment speed
The slack-off challenge of degree, while being also convenient for the treatment of unstructured data.The other present invention is traditional database with Hadoop's
The process of interaction, Hbase processing datas, is packaged into service, improves system effectiveness, is easy to call service.
Based on above-mentioned technical em- bodiments, taken, it is necessary to dispose second in the server before the reception data operation request
Business and second service；Wherein, the method for deployment first service includes：
The first model is configured in first device, first model is used for by Sqoop instruments, by traditional database
Data pick-up to Hive databases, response of the Hive databases to data operation request is realized by Hive scripts；
Specifically, the Apache Sqoop in the present embodiment are intended to assist to be carried out efficiently greatly between RDBMS and Hadoop
Data communication.User can imported into the content of traditional Relational DataBase in Hadoop with the help of Sqoop, it is also possible to
Data in Hadoop clusters are imported into traditional database
The first model in the first device is packaged into the first service, and the first service is deployed to clothes
Business device；
The method for disposing second service includes：
The second model is configured in second device, and second model is used to data are grasped by Hbase database realizings
The response that work is asked, completes the distributed operation of data；
The second model in the second device is packaged into the second service, and the second service is deployed to clothes
Business device.
Specifically, referring to Fig. 2, (1) comprises the following steps that when data are structural data：
The allocation models in device A, completes by the Sqoop instruments in Hadoop clusters, the number in relational database
Enter in Hadoop according to extracting, so as to realize data from traditional database to the movement of Hadoop clusters.
The data got in Hadoop are deposited into Hive, and according to business demand, are stored to specific table in Hive
In.
Hive scripts are write, corresponding operation is carried out to data by Hive.
In A devices, this interaction is packaged into service, facilitates different calling services.
The model for creating is disposed in device B, model is deployed on server.
(2) when data are unstructured data, comprise the following steps that：
The allocation models in device C, the operation such as distributed query of data is completed by Hbase databases.
In C device, this process is packaged into service, facilitates different calling services.
The model for creating is disposed in device D, model is deployed on server.
It should be noted that data interaction flow is packaged as corresponding service, and dispose on the server, can conveniently take
Business called side carries out service call by way of automatically or manually.When calling service automatically, being set by timer is carried out
Service is called in timing.
Based on above-mentioned any embodiment, in the present embodiment, if data corresponding with the data operation request are structure
Change data, and data corresponding with the data operation request are when needing to carry out traditional operation, and institute is responded by traditional database
State data operation request.
Specifically, referring to Fig. 3, the embodiment of the invention discloses a kind of specific data transmission scheduling method, relative to upper
One embodiment, is refined as data storage request by data operation request in the present embodiment, then the method includes：
Judge whether data to be stored are structural data；If so, directly by data storage to be stored in HDFS systems
In；
If it is not, then realizing function according to the data, judge whether the operation is traditional operation；If traditional operation, then
Data to be stored are stored in traditional Relational DataBase；If not traditional operation, that is, the number to be operated by Hadoop
According to then data to be stored are deposited into Hive.
Data transmission scheduling device provided in an embodiment of the present invention is introduced below, data transfer described below is adjusted
Degree device can be with cross-referenced with above-described data transmission scheduling method.
Referring to Fig. 4, a kind of data transmission scheduling device based on ESB provided in an embodiment of the present invention, bag
Include：
Receiver module 100, for receiving data operation request；
Judge module 200, for judging whether data corresponding with the data operation request are structural data；
When first scheduler module 300 for data corresponding with the data operation request is structural data, the is called
One service；The first service is used to, when data corresponding with the data operation request need to perform Hadoop operations, lead to
Cross data operation request described in Hive database responses；
Second scheduler module 400, during for data corresponding with the data operation request for unstructured data, calls
Second service；The second service passes through data operation request described in Hbase database responses.
Based on above-described embodiment, the present embodiment also includes：
Performing module, for data corresponding with the data operation request be structural data, and with the data
When the corresponding data of operation requests need to carry out traditional operation, the data operation request is responded by traditional database.
Based on above-described embodiment, the present embodiment also includes：
First model configuration module, for configuring the first model in first device, first model is used to pass through
Sqoop instruments, by the data pick-up in traditional database to Hive databases, Hive database logarithms are realized by Hive scripts
According to the response of operation requests；
First deployment module, for the first model in the first device to be packaged into the first service, and by institute
State first service and be deployed to server.
Based on above-described embodiment, the present embodiment also includes：
Second model configuration module, for configuring the second model in second device, second model is used to pass through
Response of the Hbase database realizings to data operation request, completes the distributed operation of data；
Second deployment module, for the second model in the second device to be packaged into the second service, and by institute
State second service and be deployed to server.
A kind of server provided in an embodiment of the present invention, including the data transmission scheduling device in above-mentioned any embodiment.
A kind of data transmission scheduling method based on ESB provided in an embodiment of the present invention, including：Receive number
According to operation requests；Judge whether data corresponding with the data operation request are structural data；If so, then calling the first clothes
Business；The first service is used to, when data corresponding with the data operation request need to perform Hadoop operations, pass through
Data operation request described in Hive database responses；If it is not, then calling second service；The second service passes through Hbase data
Storehouse responds the data operation request；
It can be seen that, in scheme, interacting for structural data and Hadoop is realized in traditional forms of enterprises's service bus, solve to pass
The problem that system database purchase scarce capacity, data analysis capabilities are not enough, improves the storage capacity and computational efficiency of data,
The treatment of unstructured data is facilitated, and service is packaged into by interaction, facilitate opening for Web Service
Send out, call, adapted to business rapid response to customer's need, improve the efficiency and scalability of system；The invention also discloses one kind
Data transmission scheduling device and server based on ESB.
Each embodiment is described by the way of progressive in this specification, and what each embodiment was stressed is and other
The difference of embodiment, between each embodiment identical similar portion mutually referring to.
The foregoing description of the disclosed embodiments, enables professional and technical personnel in the field to realize or uses the present invention.
Various modifications to these embodiments will be apparent for those skilled in the art, as defined herein
General Principle can be realized in other embodiments without departing from the spirit or scope of the present invention.Therefore, the present invention
The embodiments shown herein is not intended to be limited to, and is to fit to and principles disclosed herein and features of novelty phase one
The scope most wide for causing.
Claims (9)
1. a kind of data transmission scheduling method based on ESB, it is characterised in that including：
Receive data operation request；
Judge whether data corresponding with the data operation request are structural data；
If so, then calling first service；The first service is used to need to hold when data corresponding with the data operation request
When row Hadoop is operated, by data operation request described in Hive database responses；
If it is not, then calling second service；The second service passes through data operation request described in Hbase database responses.
2. data transmission scheduling method according to claim 1, it is characterised in that
If data corresponding with the data operation request are structural data, and data corresponding with the data operation request
When needing to carry out traditional operation, the data operation request is responded by traditional database.
3. data transmission scheduling method according to claim 1 and 2, it is characterised in that the reception data operation request
Before, also include：
The first model is configured in first device, first model is used for by Sqoop instruments, by the number in traditional database
According to extracting to Hive databases, response of the Hive databases to data operation request is realized by Hive scripts；
The first model in the first device is packaged into the first service, and the first service is deployed to service
Device.
4. data transmission scheduling method according to claim 1 and 2, it is characterised in that the reception data operation request
Before, also include：
The second model is configured in second device, second model is used for please to data manipulation by Hbase database realizings
The response asked, completes the distributed operation of data；
The second model in the second device is packaged into the second service, and the second service is deployed to service
Device.
5. a kind of data transmission scheduling device based on ESB, it is characterised in that including：
Receiver module, for receiving data operation request；
Judge module, for judging whether data corresponding with the data operation request are structural data；
First scheduler module, during for data corresponding with the data operation request for structural data, calls first service；
The first service is used for when data corresponding with the data operation request need to perform Hadoop operations, by Hive numbers
The data operation request is responded according to storehouse；
Second scheduler module, during for data corresponding with the data operation request for unstructured data, calls the second clothes
Business；The second service passes through data operation request described in Hbase database responses.
6. data transmission scheduling device according to claim 5, it is characterised in that also include：
Performing module, for data corresponding with the data operation request be structural data, and with the data manipulation
When asking corresponding data and needing to carry out traditional operation, the data operation request is responded by traditional database.
7. the data transmission scheduling device according to claim 5 or 6, it is characterised in that also include：
First model configuration module, for configuring the first model in first device, first model is used to pass through Sqoop works
Tool, by the data pick-up in traditional database to Hive databases, realizes Hive databases to data manipulation by Hive scripts
The response of request；
First deployment module, for the first model in the first device to be packaged into the first service, and by described
One service arrangement is to server.
8. the data transmission scheduling device according to claim 5 or 6, it is characterised in that also include：
Second model configuration module, for configuring the second model in second device, second model is used to pass through Hbase numbers
The response to data operation request is realized according to storehouse, the distributed operation of data is completed；
Second deployment module, for the second model in the second device to be packaged into the second service, and by described
Two service arrangements are to server.
9. a kind of server, it is characterised in that including the data transmission scheduling dress as described in any one in claim 5-8
Put.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611217919.1A CN106844496B (en) | 2016-12-26 | 2016-12-26 | Data transmission scheduling method and device based on enterprise service bus and server |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611217919.1A CN106844496B (en) | 2016-12-26 | 2016-12-26 | Data transmission scheduling method and device based on enterprise service bus and server |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106844496A true CN106844496A (en) | 2017-06-13 |
| CN106844496B CN106844496B (en) | 2020-04-10 |
Family
ID=59136595
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611217919.1A Active CN106844496B (en) | 2016-12-26 | 2016-12-26 | Data transmission scheduling method and device based on enterprise service bus and server |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106844496B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103678665A (en) * | 2013-12-24 | 2014-03-26 | 焦点科技股份有限公司 | Heterogeneous large data integration method and system based on data warehouses |
| US20140172809A1 (en) * | 2012-12-13 | 2014-06-19 | William Gardella | Hadoop access via hadoop interface services based on function conversion |
| CN104820670A (en) * | 2015-03-13 | 2015-08-05 | 国家电网公司 | Method for acquiring and storing big data of power information |
| CN105243155A (en) * | 2015-10-29 | 2016-01-13 | 贵州电网有限责任公司电力调度控制中心 | Big data extracting and exchanging system |
| CN105468720A (en) * | 2015-11-20 | 2016-04-06 | 北京锐安科技有限公司 | Method for integrating distributed data processing systems, corresponding systems and data processing method |
| CN105760505A (en) * | 2016-02-23 | 2016-07-13 | 浪潮软件集团有限公司 | Hive-based historical data analysis and archiving method |
-
2016
- 2016-12-26 CN CN201611217919.1A patent/CN106844496B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140172809A1 (en) * | 2012-12-13 | 2014-06-19 | William Gardella | Hadoop access via hadoop interface services based on function conversion |
| CN103678665A (en) * | 2013-12-24 | 2014-03-26 | 焦点科技股份有限公司 | Heterogeneous large data integration method and system based on data warehouses |
| CN104820670A (en) * | 2015-03-13 | 2015-08-05 | 国家电网公司 | Method for acquiring and storing big data of power information |
| CN105243155A (en) * | 2015-10-29 | 2016-01-13 | 贵州电网有限责任公司电力调度控制中心 | Big data extracting and exchanging system |
| CN105468720A (en) * | 2015-11-20 | 2016-04-06 | 北京锐安科技有限公司 | Method for integrating distributed data processing systems, corresponding systems and data processing method |
| CN105760505A (en) * | 2016-02-23 | 2016-07-13 | 浪潮软件集团有限公司 | Hive-based historical data analysis and archiving method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106844496B (en) | 2020-04-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102880503B (en) | Data analysis system and data analysis method | |
| CN100531055C (en) | Data synchronous system and its method | |
| US8775626B2 (en) | Using templates to configure cloud resources | |
| CN104391918B (en) | The implementation method of distributed networks database query priority management based on equity deployment | |
| CN106599711A (en) | Database access control method and device | |
| CN108880887A (en) | Accompany and attend to robot cloud service system and method based on micro services | |
| CN106651633A (en) | Power utilization information acquisition system and method based on big data technology | |
| CN112492003B (en) | Cloud network convergence service processing method and device based on service orchestrator | |
| CN105635283A (en) | Organization and management and using method and system for cloud manufacturing service | |
| CN103019651A (en) | Parallel processing method and device for complex tasks | |
| CN102929961A (en) | Data processing method and device thereof based on building quick data staging channel | |
| CN106814975A (en) | A kind of memory management method and its system for polymorphic type storage device | |
| CN104536963B (en) | A kind of dispatching method and system of storing process | |
| CN102902669A (en) | Distribution information capturing method based on internet system | |
| CN101556586A (en) | Method, system and device of automatic data collection | |
| CN103902389A (en) | Equipment management method supporting bidirectional communication based on android platform | |
| CN109074287A (en) | Infrastructure resources state | |
| CN109815254A (en) | Cross-region method for scheduling task and system based on big data | |
| CN107203560A (en) | Database, multiple database operation transaction consistency ensuring method and system | |
| CN102364889A (en) | Shared pool resource charging method and device | |
| CN108920274A (en) | Performance optimization and device for image processing server end | |
| CN105637904B (en) | For the service interworking for the communication session for being related to wireless terminal and server apparatus | |
| CN113515363B (en) | Special-shaped task high-concurrency multi-level data processing system dynamic scheduling platform | |
| CN102629345B (en) | Chain type links up collaboration method, Apparatus and system | |
| CN106844496A (en) | Data transmission scheduling method, device and server based on ESB |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |