CN106844496B - Data transmission scheduling method and device based on enterprise service bus and server - Google Patents
Data transmission scheduling method and device based on enterprise service bus and server Download PDFInfo
- Publication number
- CN106844496B CN106844496B CN201611217919.1A CN201611217919A CN106844496B CN 106844496 B CN106844496 B CN 106844496B CN 201611217919 A CN201611217919 A CN 201611217919A CN 106844496 B CN106844496 B CN 106844496B
- Authority
- CN
- China
- Prior art keywords
- data
- service
- operation request
- database
- model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data transmission scheduling method, a device and a server based on an enterprise service bus, comprising the following steps: judging whether the data corresponding to the data operation request is structured data or not; if yes, calling the first service; the first service is used for responding to the data operation request through the Hive database when data corresponding to the data operation request needs to execute Hadoop operation; and if not, calling a second service, and responding to the data operation request through the Hbase database. The interaction between the structured data and the Hadoop is realized in the traditional enterprise Service bus, the problems of insufficient energy storage capacity and insufficient data analysis capacity of the traditional database are solved, the storage capacity and the calculation efficiency of the data are improved, the processing of the unstructured data is facilitated, the development and the calling of Web Service are facilitated by packaging the interaction process into Service, the requirement of quick Service response is met, and the efficiency and the expandability of the system are improved.
Description
Technical Field
The invention relates to the technical field of data transmission, in particular to a data transmission scheduling method and device based on an enterprise service bus and a server.
Background
The enterprise service bus is a product of the combination of traditional middleware technology and technologies such as XML, Web services and the like. The enterprise service bus provides the most basic connection center in the network and is an essential element for constructing the enterprise nervous system. Functionally, the enterprise service bus provides an event-driven and document-oriented processing model, as well as a distributed operation management mechanism that supports content-based routing and filtering, has complex data transfer capabilities, and can provide a range of standard interfaces. The amount of information in the enterprise service bus is increasing rapidly, and more enterprises find that the existing traditional data storage and processing mode cannot cope with the increase speed of data and cannot meet the requirement of users on data processing.
Therefore, it is necessary for those skilled in the art to solve the problem of insufficient energy storage capacity and data analysis capability of the conventional database and improve the system processing efficiency.
Disclosure of Invention
The invention aims to provide a data transmission scheduling method, a data transmission scheduling device and a data transmission scheduling server based on an enterprise service bus, so as to solve the problems of insufficient energy storage capacity and data analysis capacity of a traditional database and improve the system processing efficiency.
In order to achieve the above purpose, the embodiment of the present invention provides the following technical solutions:
a data transmission scheduling method based on an enterprise service bus comprises the following steps:
receiving a data operation request;
judging whether the data corresponding to the data operation request is structured data or not;
if yes, calling the first service; the first service is used for responding to the data operation request through a Hive database when data corresponding to the data operation request needs to execute a Hadoop operation;
if not, calling a second service; the second service responds to the data operation request through the Hbase database.
And if the data corresponding to the data operation request is structured data and the data corresponding to the data operation request needs to be subjected to traditional operation, responding to the data operation request through a traditional database.
Before receiving the data operation request, the method further includes:
configuring a first model in a first device, wherein the first model is used for extracting data in a traditional database to a Hive database through a Sqoop tool, and realizing the response of the Hive database to a data operation request through a Hive script;
packaging the first model in the first device into the first service, and deploying the first service to a server.
Before receiving the data operation request, the method further includes:
configuring a second model in a second device, wherein the second model is used for realizing the response to the data operation request through an Hbase database and completing the distributed operation of the data;
packaging the second model in the second device into the second service, and deploying the second service to a server.
A data transmission scheduling device based on an enterprise service bus comprises:
the receiving module is used for receiving a data operation request;
the judging module is used for judging whether the data corresponding to the data operation request is structured data or not;
the first scheduling module is used for calling a first service when the data corresponding to the data operation request is structured data; the first service is used for responding to the data operation request through a Hive database when data corresponding to the data operation request needs to execute a Hadoop operation;
the second scheduling module is used for calling a second service when the data corresponding to the data operation request is unstructured data; the second service responds to the data operation request through the Hbase database.
Wherein, still include:
and the execution module is used for responding to the data operation request through a traditional database when the data corresponding to the data operation request is structured data and the data corresponding to the data operation request needs to be subjected to traditional operation.
Wherein, still include:
the device comprises a first model configuration module, a second model configuration module and a third model configuration module, wherein the first model configuration module is used for configuring a first model in a first device, the first model is used for extracting data in a traditional database to a Hive database through a Sqoop tool, and the response of the Hive database to a data operation request is realized through a Hive script;
the first deployment module is used for packaging the first model in the first device into the first service and deploying the first service to a server.
Wherein, still include:
the second model configuration module is used for configuring a second model in a second device, and the second model is used for realizing the response to the data operation request through an Hbase database and completing the distributed operation of the data;
and the second deployment module is used for packaging the second model in the second device into the second service and deploying the second service to the server.
A server comprises any one of the data transmission scheduling devices.
According to the above scheme, the data transmission scheduling method based on the enterprise service bus provided by the embodiment of the invention comprises the following steps: receiving a data operation request; judging whether the data corresponding to the data operation request is structured data or not; if yes, calling the first service; the first service is used for responding to the data operation request through a Hive database when data corresponding to the data operation request needs to execute a Hadoop operation; if not, calling a second service; the second service responds to the data operation request through an Hbase database;
therefore, in the scheme, the interaction between the structured data and the Hadoop is realized in the traditional enterprise Service bus, the problems of insufficient energy storage capacity and insufficient data analysis capacity of the traditional database are solved, the storage capacity and the calculation efficiency of the data are improved, the processing of the unstructured data is facilitated, the development and the calling of Web Service are facilitated by packaging the interaction process into Service, the quick Service response requirement is met, and the efficiency and the expandability of the system are improved; the invention also discloses a data transmission scheduling device and a server based on the enterprise service bus.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a schematic flowchart of a data transmission scheduling method based on an enterprise service bus according to an embodiment of the present invention;
FIG. 2 is a schematic diagram illustrating an operation flow of a scheduling mechanism for data transmission based on a Hadoop cluster according to an embodiment of the present invention;
FIG. 3 is a schematic flow chart of a data storage method according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a data transmission scheduling apparatus based on an enterprise service bus according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The embodiment of the invention discloses a data transmission scheduling method, a data transmission scheduling device and a data transmission scheduling server based on an enterprise service bus, which are used for solving the problems of insufficient energy storage capacity and data analysis capacity of a traditional database and improving the system processing efficiency.
Referring to fig. 1, an embodiment of the present invention provides a data transmission scheduling method based on an enterprise service bus, including:
s101, receiving a data operation request;
s102, judging whether the data corresponding to the data operation request is structured data or not;
if yes, executing S103; if not, executing S104;
s103, calling a first service; the first service is used for responding to the data operation request through a Hive database when data corresponding to the data operation request needs to execute a Hadoop operation;
specifically, in the scheme, Hive is a data warehouse infrastructure established on Hadoop, which can regard massive structured data in Hive as individual tables, and actually, the data are stored in HDFS in a distributed manner, which provides a complete SQL query function and can convert SQL statements into Map/Reduce tasks.
S104, calling a second service; the second service responds to the data operation request through the Hbase database.
Specifically, HBase is a sub-item of the Hadoop item of Apache. HBase is different from a general relational database, and is a database suitable for unstructured data storage; another difference is that HBase is based on a column rather than a row based pattern.
Therefore, the embodiment discloses a scheduling mechanism for data transmission by using a Hadoop cluster in an enterprise service bus, which realizes interaction between structured data and Hadoop in a traditional enterprise service bus, and is convenient for processing unstructured data while coping with the challenges of large data volume and slow processing speed. In addition, the invention packages the interaction process of the traditional database and Hadoop and the process of processing data by Hbase into service, thereby improving the system efficiency and facilitating the calling of the service.
Based on the technical embodiment, before the data operation request is received, a second service and a second service need to be deployed in the server; the method for deploying the first service comprises the following steps:
configuring a first model in a first device, wherein the first model is used for extracting data in a traditional database to a Hive database through a Sqoop tool, and realizing the response of the Hive database to a data operation request through a Hive script;
specifically, Apache Sqoop in this embodiment is intended to assist efficient big data communication between the RDBMS and Hadoop. Users can import the content of the traditional relational database into the Hadoop with the help of Sqoop, and can import the data in the Hadoop cluster into the traditional database
Packaging a first model in the first device into the first service and deploying the first service to a server;
the method for deploying the second service comprises the following steps:
configuring a second model in a second device, wherein the second model is used for realizing the response to the data operation request through an Hbase database and completing the distributed operation of the data;
packaging the second model in the second device into the second service, and deploying the second service to a server.
Specifically, referring to fig. 2, (1) when the data is structured data, the specific steps are as follows:
and configuring a model in the device A, and extracting data in the correlation coefficient database into the Hadoop through an Sqoop tool in the Hadoop cluster, so that the data can be moved from the traditional database to the Hadoop cluster.
And storing the data obtained from the Hadoop into the Hive, and storing the data into a specific table in the Hive according to the service requirement.
Writing Hive script and carrying out corresponding operation on the data through Hive.
In the device A, the interactive process is packaged into service, so that different services can be conveniently called.
The created model is deployed in the device B, and the model is deployed to the server.
(2) When the data is unstructured, the specific steps are as follows:
and configuring a model in the device C, and completing operations such as distributed query of data through an Hbase database.
In the device C, the process is packaged into a service, so that different services can be conveniently called.
The created model is deployed in the device D, and the model is deployed to the server.
It should be noted that, the data interaction process is packaged as the corresponding service and deployed on the server, which is convenient for the service invoker to invoke the service automatically or manually. And when the service is automatically called, the service is called regularly through timer setting.
Based on any of the above embodiments, in this embodiment, if the data corresponding to the data operation request is structured data and the data corresponding to the data operation request needs to be subjected to a conventional operation, the data operation request is responded to through a conventional database.
Specifically, referring to fig. 3, an embodiment of the present invention discloses a specific data transmission scheduling method, and with respect to the previous embodiment, in this embodiment, a data operation request is detailed as a data storage request, and the method includes:
judging whether the data to be stored is structured data or not; if yes, directly storing the data to be stored in the HDFS system;
if not, judging whether the operation is the traditional operation or not according to the realization function of the data; if the operation is traditional, storing the data to be stored in a traditional relational database; and if the operation is not the traditional operation, namely the data to be subjected to the Hadoop operation, storing the data to be stored into the Hive.
In the following, the data transmission scheduling apparatus provided in the embodiment of the present invention is introduced, and the data transmission scheduling apparatus described below and the data transmission scheduling method described above may be referred to each other.
Referring to fig. 4, an embodiment of the present invention provides an apparatus for scheduling data transmission based on an enterprise service bus, including:
a receiving module 100, configured to receive a data operation request;
a judging module 200, configured to judge whether data corresponding to the data operation request is structured data;
the first scheduling module 300 is configured to invoke a first service when the data corresponding to the data operation request is structured data; the first service is used for responding to the data operation request through a Hive database when data corresponding to the data operation request needs to execute a Hadoop operation;
a second scheduling module 400, configured to invoke a second service when the data corresponding to the data operation request is unstructured data; the second service responds to the data operation request through the Hbase database.
Based on the above embodiment, this embodiment further includes:
and the execution module is used for responding to the data operation request through a traditional database when the data corresponding to the data operation request is structured data and the data corresponding to the data operation request needs to be subjected to traditional operation.
Based on the above embodiment, this embodiment further includes:
the device comprises a first model configuration module, a second model configuration module and a third model configuration module, wherein the first model configuration module is used for configuring a first model in a first device, the first model is used for extracting data in a traditional database to a Hive database through a Sqoop tool, and the response of the Hive database to a data operation request is realized through a Hive script;
the first deployment module is used for packaging the first model in the first device into the first service and deploying the first service to a server.
Based on the above embodiment, this embodiment further includes:
the second model configuration module is used for configuring a second model in a second device, and the second model is used for realizing the response to the data operation request through an Hbase database and completing the distributed operation of the data;
and the second deployment module is used for packaging the second model in the second device into the second service and deploying the second service to the server.
The server provided by the embodiment of the present invention includes the data transmission scheduling apparatus in any of the above embodiments.
The embodiment of the invention provides a data transmission scheduling method based on an enterprise service bus, which comprises the following steps: receiving a data operation request; judging whether the data corresponding to the data operation request is structured data or not; if yes, calling the first service; the first service is used for responding to the data operation request through a Hive database when data corresponding to the data operation request needs to execute a Hadoop operation; if not, calling a second service; the second service responds to the data operation request through an Hbase database;
therefore, in the scheme, the interaction between the structured data and the Hadoop is realized in the traditional enterprise Service bus, the problems of insufficient energy storage capacity and insufficient data analysis capacity of the traditional database are solved, the storage capacity and the calculation efficiency of the data are improved, the processing of the unstructured data is facilitated, the development and the calling of Web Service are facilitated by packaging the interaction process into Service, the quick Service response requirement is met, and the efficiency and the expandability of the system are improved; the invention also discloses a data transmission scheduling device and a server based on the enterprise service bus.
The embodiments in the present description are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (5)
1. A data transmission scheduling method based on an enterprise service bus is characterized by comprising the following steps:
receiving a data operation request;
judging whether the data corresponding to the data operation request is structured data or not;
if yes, calling the first service; the first service is used for responding to the data operation request through a Hive database when data corresponding to the data operation request needs to execute a Hadoop operation;
if not, calling a second service; the second service responds to the data operation request through an Hbase database;
before receiving the data operation request, the method further includes: configuring a first model in a first device, wherein the first model is used for extracting data in a traditional relational database to a Hive database through a Sqoop tool, and realizing the response of the Hive database to a data operation request through a Hive script; packaging a first model in the first device into the first service and deploying the first service to a server;
configuring a second model in a second device, wherein the second model is used for realizing the response to the data operation request through an Hbase database and completing the distributed operation of the data; packaging the second model in the second device into the second service, and deploying the second service to a server.
2. The data transmission scheduling method of claim 1,
and if the data corresponding to the data operation request is structured data and the data corresponding to the data operation request needs to be subjected to traditional operation, responding to the data operation request through a traditional database.
3. A data transmission scheduling apparatus based on an enterprise service bus, comprising:
the receiving module is used for receiving a data operation request;
the judging module is used for judging whether the data corresponding to the data operation request is structured data or not;
the first scheduling module is used for calling a first service when the data corresponding to the data operation request is structured data; the first service is used for responding to the data operation request through a Hive database when data corresponding to the data operation request needs to execute a Hadoop operation;
the second scheduling module is used for calling a second service when the data corresponding to the data operation request is unstructured data; the second service responds to the data operation request through an Hbase database;
the first model configuration module is used for configuring a first model in a first device, wherein the first model is used for extracting data in a traditional relational database to a Hive database through a Sqoop tool, and realizing the response of the Hive database to a data operation request through a Hive script;
a first deployment module, configured to package a first model in the first device into the first service, and deploy the first service to a server;
the second model configuration module is used for configuring a second model in a second device, and the second model is used for realizing the response to the data operation request through an Hbase database and completing the distributed operation of the data;
and the second deployment module is used for packaging the second model in the second device into the second service and deploying the second service to the server.
4. The data transmission scheduling apparatus of claim 3, further comprising:
and the execution module is used for responding to the data operation request through a traditional database when the data corresponding to the data operation request is structured data and the data corresponding to the data operation request needs to be subjected to traditional operation.
5. A server, characterized in that it comprises a data transmission scheduling device according to claim 3 or 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611217919.1A CN106844496B (en) | 2016-12-26 | 2016-12-26 | Data transmission scheduling method and device based on enterprise service bus and server |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611217919.1A CN106844496B (en) | 2016-12-26 | 2016-12-26 | Data transmission scheduling method and device based on enterprise service bus and server |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106844496A CN106844496A (en) | 2017-06-13 |
| CN106844496B true CN106844496B (en) | 2020-04-10 |
Family
ID=59136595
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611217919.1A Active CN106844496B (en) | 2016-12-26 | 2016-12-26 | Data transmission scheduling method and device based on enterprise service bus and server |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106844496B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103678665A (en) * | 2013-12-24 | 2014-03-26 | 焦点科技股份有限公司 | Heterogeneous large data integration method and system based on data warehouses |
| CN104820670A (en) * | 2015-03-13 | 2015-08-05 | 国家电网公司 | Method for acquiring and storing big data of power information |
| CN105243155A (en) * | 2015-10-29 | 2016-01-13 | 贵州电网有限责任公司电力调度控制中心 | Big data extracting and exchanging system |
| CN105468720A (en) * | 2015-11-20 | 2016-04-06 | 北京锐安科技有限公司 | Method for integrating distributed data processing systems, corresponding systems and data processing method |
| CN105760505A (en) * | 2016-02-23 | 2016-07-13 | 浪潮软件集团有限公司 | Hive-based historical data analysis and archiving method |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9031925B2 (en) * | 2012-12-13 | 2015-05-12 | Sap Se | Hadoop access via hadoop interface services based on function conversion |
-
2016
- 2016-12-26 CN CN201611217919.1A patent/CN106844496B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103678665A (en) * | 2013-12-24 | 2014-03-26 | 焦点科技股份有限公司 | Heterogeneous large data integration method and system based on data warehouses |
| CN104820670A (en) * | 2015-03-13 | 2015-08-05 | 国家电网公司 | Method for acquiring and storing big data of power information |
| CN105243155A (en) * | 2015-10-29 | 2016-01-13 | 贵州电网有限责任公司电力调度控制中心 | Big data extracting and exchanging system |
| CN105468720A (en) * | 2015-11-20 | 2016-04-06 | 北京锐安科技有限公司 | Method for integrating distributed data processing systems, corresponding systems and data processing method |
| CN105760505A (en) * | 2016-02-23 | 2016-07-13 | 浪潮软件集团有限公司 | Hive-based historical data analysis and archiving method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106844496A (en) | 2017-06-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111400326B (en) | Smart city data management system and method thereof | |
| CN102880503B (en) | Data analysis system and data analysis method | |
| TWI549080B (en) | The method, system and device for sending information of category information | |
| CN109857801B (en) | Intelligent irrigation area e platform based on four-layer architecture and data processing method | |
| CN105550268A (en) | Big data process modeling analysis engine | |
| CN110891094A (en) | Terminal software supporting platform based on middle platform | |
| CN103605764A (en) | Web crawler system and web crawler multitask executing and scheduling method | |
| CN107103064B (en) | Data statistical method and device | |
| CN103631870A (en) | System and method used for large-scale distributed data processing | |
| CN106599711A (en) | Database access control method and device | |
| WO2019047441A1 (en) | Communication optimization method and system | |
| CN110968579B (en) | Execution plan generation and execution method, database engine and storage medium | |
| CN102932448A (en) | Distributed network crawler URL (uniform resource locator) duplicate removal system and method | |
| CN107870982B (en) | Data processing method, system and computer readable storage medium | |
| CN102567378A (en) | Information retrieval system based on heterogeneous data | |
| CN103823846A (en) | Method for storing and querying big data on basis of graph theories | |
| CN112367354A (en) | Intelligent scheduling system and scheduling method for cloud-edge resource graph | |
| CN102750368A (en) | High-speed importing method of cluster data in data base | |
| CN114500676A (en) | Information interaction method and device among industrial internet devices and storage medium | |
| CN106844496B (en) | Data transmission scheduling method and device based on enterprise service bus and server | |
| CN112417225A (en) | Joint query method and system for multi-source heterogeneous data | |
| CN101692737B (en) | Light weight data synchronization system and method | |
| CN112035466A (en) | Block chain query external index development framework | |
| CN116069462A (en) | Big data DAG task flow scheduling method, system and storage medium | |
| CN105208095A (en) | Filtering service deployment recommendation device and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |