CN118364033A - Data processing method and device among database clusters and related equipment - Google Patents
Data processing method and device among database clusters and related equipment Download PDFInfo
- Publication number
- CN118364033A CN118364033A CN202410789311.4A CN202410789311A CN118364033A CN 118364033 A CN118364033 A CN 118364033A CN 202410789311 A CN202410789311 A CN 202410789311A CN 118364033 A CN118364033 A CN 118364033A
- Authority
- CN
- China
- Prior art keywords
- data
- source
- database
- service gateway
- receiving
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title abstract description 14
- 238000000034 method Methods 0.000 claims abstract description 40
- 238000012545 processing Methods 0.000 claims abstract description 38
- 238000004590 computer program Methods 0.000 claims description 22
- 238000004458 analytical method Methods 0.000 abstract description 14
- 230000006870 function Effects 0.000 description 10
- 230000005540 biological transmission Effects 0.000 description 9
- 238000010586 diagram Methods 0.000 description 5
- 230000008569 process Effects 0.000 description 4
- 230000008859 change Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000007405 data analysis Methods 0.000 description 2
- 238000013480 data collection Methods 0.000 description 2
- 230000008676 import Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- OKTJSMMVPCPJKN-UHFFFAOYSA-N Carbon Chemical compound [C] OKTJSMMVPCPJKN-UHFFFAOYSA-N 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 229910021389 graphene Inorganic materials 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a data processing method, a device and related equipment among database clusters. The method is applied to the technical field of data processing, and comprises the following steps when applied to a target database end: sending a data pulling request to a service gateway and receiving a data source end list structure returned by the service gateway; according to the data source end table structure, a local temporary table is created, and data of a corresponding type is obtained from a source database end according to the data source type indicated in the data pulling request; and pulling corresponding data from the source database end to a local temporary table so as to finish data processing. By the scheme, data exchange and analysis can be performed across the database cluster.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a method and an apparatus for processing data between database clusters, and related devices.
Background
Because the large integrated service platform generally comprises a plurality of subsystems, various exchange protocols and business flow correlations exist among the systems, and if effective processing is not performed, a large amount of data generated along with the business process cannot be uniformly managed. Therefore, the data sharing and data exchanging system is constructed to avoid a series of problems such as repeated data collection, data redundancy, asynchronous shared data updating, inaccurate data, unsmooth information communication, inconsistent data formats and the like among the systems. All information data are stored and planned in a unified way through the system, so that the data of each subsystem can be shared among different subsystems.
However, the current data sharing or exchange is used in the same type of database, and a method capable of satisfying the data exchange and analysis between different database clusters, namely, performing data exchange and analysis across database clusters is needed.
Disclosure of Invention
In view of the foregoing, it is necessary to provide a data processing method, apparatus and related devices capable of implementing data exchange and analysis across database clusters.
In a first aspect, the present application provides a method for processing data between database clusters, applied to a target database end, where the method includes:
sending a data pulling request to a service gateway and receiving a data source end list structure returned by the service gateway;
According to the data source end table structure, a local temporary table is created, and data of a corresponding type is obtained from a source database end according to the data source type indicated in the data pulling request;
and pulling the corresponding data from the source database end to the local temporary table to finish data processing.
In one embodiment, sending a data pull request to a service gateway and receiving a data source table structure returned by the service gateway includes:
in the case where the scene is an isomorphic scene:
sending a data pulling request to a service gateway, wherein the data pulling request is used for indicating the service gateway to send a Structured Query Language (SQL) of a data source end table structure to a source database end and receiving the data source end table structure from the source database end;
a data source end table structure is received from a service gateway.
In one embodiment, sending a data pull request to a service gateway and receiving a data source table structure returned by the service gateway includes:
in the case where the scene is a heterogeneous scene:
Sending a data pulling request to a service gateway, wherein the data pulling request is used for indicating the service gateway to send a Structured Query Language (SQL) of a data source end table structure to a source database end and receiving the data source end table structure from the source database end; judging the type of a source database terminal according to the data pulling request, sending a Structured Query Language (SQL) of a data source terminal list structure to the source database terminal, formatting the heterogeneous database terminal into a format corresponding to the data source terminal list structure at the source database terminal, and receiving the data source terminal list structure from the source database terminal;
a data source end table structure is received from a service gateway.
In one embodiment, obtaining data of a corresponding type from a source database according to a data source type indicated in a data pull request includes:
in the case where the scene is an isomorphic scene:
Judging whether the data nodes of the target database end and the source database end can communicate or not;
If yes, directly acquiring a data file of a corresponding type from a source database terminal according to the data source type indicated in the data pulling request;
if not, the control service gateway executes a query statement according to the type of the source database terminal to acquire data, and formats the data into the data of the corresponding type.
In one embodiment, obtaining data of a corresponding type from a source database according to a data source type indicated in a data pull request includes:
in the case where the scene is a heterogeneous scene:
and the control service gateway executes a query statement according to the type of the source database end to acquire data, and formats the data into data of a corresponding type.
In a second aspect, the present application further provides a data processing device between database clusters, applied to a target database end side, where the device includes:
The receiving module is used for sending a data pulling request to the service gateway and receiving a data source end list structure returned by the service gateway;
The acquisition module is used for creating a local temporary table according to the data source end table structure and acquiring corresponding type data from a source database end according to the data source type indicated in the data pulling request;
and the pulling module is used for pulling the corresponding data from the source database end to the local temporary table so as to finish data processing.
In a third aspect, the present application also provides a computer device comprising a memory and a processor, the memory storing a computer program, the processor executing the computer program to perform the steps of the method for processing data between database clusters described above.
In a fourth aspect, the present application also provides a computer readable storage medium having a computer program stored thereon, which when executed by a processor, implements the steps of the method for processing data between database clusters described above.
In a fifth aspect, the application also provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the method for processing data between database clusters described above.
The data processing method among the database clusters is used for sending a data pulling request to the service gateway when being applied to the target database end and receiving a data source end table structure returned by the service gateway; according to the data source end table structure, a local temporary table is created, and data of a corresponding type is obtained from a source database end according to the data source type indicated in the data pulling request; and pulling the corresponding data from the source database end to the local temporary table to finish data processing. The data processing method among the database clusters can realize data exchange and analysis across the database clusters.
Drawings
FIG. 1 is a flow chart of a method of data processing among database clusters in one embodiment;
FIG. 2 is a flow diagram of a data source table structure for sending a data pull request to a service gateway and receiving a return from the service gateway in one embodiment;
FIG. 3 is a flow chart of acquiring corresponding types of data from a source database in one embodiment;
FIG. 4 is a block diagram of a data processing apparatus between clusters of data databases in one embodiment;
Fig. 5 is an internal structural diagram of a computer device in one embodiment.
Detailed Description
The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.
Because the large integrated service platform generally comprises a plurality of subsystems, various exchange protocols and business flow correlations exist among the systems, and if effective processing is not performed, a large amount of data generated along with the business process cannot be uniformly managed. Therefore, the data sharing and data exchanging system is constructed to avoid a series of problems such as repeated data collection, data redundancy, asynchronous shared data updating, inaccurate data, unsmooth information communication, inconsistent data formats and the like among the systems. All information data are stored and planned in a unified way through the system, so that the data of each subsystem can be shared among different subsystems. However, the current data sharing or exchange is used in the same type of database, and a method capable of satisfying the data exchange and analysis between different database clusters, namely, performing data exchange and analysis across database clusters is needed. Based on this, the embodiment of the application provides a data processing method between database clusters, so as to improve the technical problems.
In one embodiment, fig. 1 is a method for processing data between database clusters according to an embodiment of the present application, and is described by taking application of the method to a target database end as an example, the method includes the following steps:
S101, sending a data pulling request to a service gateway and receiving a data source end list structure returned by the service gateway.
Alternatively, the transparent gateway is a separate service process written in Java language and can be deployed separately at a separate node. The transparent gateway provides data transmission functions among GBase8a clusters, including online data import functions from third party databases such as Oracle, mySQL, etc. into the GBase8a cluster databases.
Meanwhile, the transparent gateway provides a data transmission function between GBase8a clusters, including an online data import function from a third party database such as Oracle, mySQL and the like into the GBase8a cluster database.
S102, a local temporary table is created according to the data source end table structure, and data of a corresponding type is acquired from a source database end according to the data source type indicated in the data pulling request.
Alternatively, the scenarios of data transmission are divided into two scenarios, where the same type of scenario, such as GBase8A and GBase8A, is referred to as isomorphic scenarios, and GBase8A and other databases (mysql/oracle/teradata/oceanbase) are referred to as heterogeneous scenarios below.
For isomorphic scenes, the GBase8a data file formats are cross-version compatible, exchange can be carried out among different clusters through a data file transmission mode, and the transparent gateway is used for connecting the two clusters to communicate and coordinating the data transmission function of the data nodes; for the heterogeneous scene, the data file formats are not compatible, the transparent gateway needs to acquire a result set of heterogeneous data, convert the result set into SQL sentences and send the SQL sentences to the GBase8A database, and therefore the data transmission function is achieved. And the GBase8a data is also sent to other heterogeneous databases to be collected through a transparent gateway, and is converted into different SQL according to different types of databases and sent to corresponding databases, so that the online data transmission is completed.
Optionally, the embodiment of the application can meet the data transmission requirements among different GBase8a database clusters, including online data exchange with third party databases (such as Oracle, mySQL, etc.), and through the scheme, the user can realize the database requirements of data sharing and exchange among the database clusters. Therefore, the embodiment of the application realizes the online data transmission function of the cross database cluster and solves the difficult problem of data exchange between different database systems; the method has good cross-platform compatibility and can be suitable for database clusters in different environments.
S103, pulling corresponding data from the source database end to the local temporary table to finish data processing.
Optionally, the data on-line analysis function of the embodiment of the application: the transparent gateway provides an online analysis function of the GBase8a database by using external database data, and the select statement can directly use the transparent gateway to establish connection with other external databases.
Optionally, the scenes of the data analysis are classified into isomorphic scenes and heterogeneous scenes.
Under an isomorphic scene such as GBase8a, the GBase8a firstly acquires metadata information of a data source related table through a transparent gateway, creates a temporary table with the same table structure at a local target end, and pulls source data into the temporary table, so that the online data query analysis function is completed.
Under the heterogeneous scene, the isomorphic scene is similar, the table structure information of the heterogeneous data source is obtained first, the data type and the table structure of the heterogeneous database are possibly different from GBase8a, the transparent gateway performs formatting processing after obtaining the table structure of the data source end, a temporary table is built at the target end, the source end data is transmitted into the temporary table, and the online data analysis work is completed at the target end.
The data processing method among the database clusters is used for sending a data pulling request to the service gateway when being applied to the target database end and receiving a data source end table structure returned by the service gateway; according to the data source end table structure, a local temporary table is created, and data of a corresponding type is obtained from a source database end according to the data source type indicated in the data pulling request; and pulling the corresponding data from the source database end to the local temporary table to finish data processing. The data processing method among the database clusters can realize data exchange and analysis across the database clusters.
On the basis of the above embodiment, the steps of sending a data pull request to the service gateway and receiving the data source table structure returned by the service gateway through fig. 2 are decomposed and refined. As shown in fig. 2, the method comprises the following implementation procedures:
in the case where the scene is an isomorphic scene:
S201, a data pulling request is sent to a service gateway, and the data pulling request is used for instructing the service gateway to send a structured query language SQL of a data source end table structure to a source database end and receiving the data source end table structure from the source database end.
S202, a data source end table structure is received from a service gateway.
Meanwhile, in the case where the scene is a heterogeneous scene:
Sending a data pulling request to a service gateway, wherein the data pulling request is used for indicating the service gateway to send a Structured Query Language (SQL) of a data source end table structure to a source database end and receiving the data source end table structure from the source database end; judging the type of a source database terminal according to the data pulling request, sending a Structured Query Language (SQL) of a data source terminal list structure to the source database terminal, formatting the heterogeneous database terminal into a format corresponding to the data source terminal list structure at the source database terminal, and receiving the data source terminal list structure from the source database terminal.
It can be understood that in this embodiment, a possible implementation manner of sending a data pulling request to the service gateway and receiving a data source table structure returned by the service gateway is provided, and the method is simple and easy to operate, and can implement data exchange and analysis across database clusters.
On the basis of the above embodiment, the step of acquiring the data of the corresponding type from the source database side is decomposed and refined by fig. 3. As shown in fig. 3, the method comprises the following implementation procedures:
in the case where the scene is an isomorphic scene:
S301, judging whether the data nodes of the target database end and the source database end can communicate.
S302, if yes, directly acquiring the data file of the corresponding type from the source database terminal according to the data source type indicated in the data pulling request.
S303, if not, the control service gateway executes a query statement according to the type of the source database terminal to acquire data, and formats the data into the data of the corresponding type.
Meanwhile, in the case where the scene is a heterogeneous scene: and the control service gateway executes a query statement according to the type of the source database end to acquire data, and formats the data into data of a corresponding type.
It can be appreciated that in this embodiment, a possible implementation manner of acquiring the data of the corresponding type from the source database is provided, and the method is simple and easy to operate, and can implement data exchange and analysis across database clusters.
It should be understood that, although the steps in the flowcharts related to the above embodiments are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.
Based on the same inventive concept, the embodiment of the application also provides a data processing device between database clusters for realizing the data processing method between database clusters. The implementation of the solution provided by the device is similar to the implementation described in the above method, so the specific limitation in the embodiments of the data processing device between one or more database clusters provided below may refer to the limitation of the data processing method between database clusters hereinabove, and will not be repeated herein.
In one embodiment, a block diagram of a data processing apparatus between database clusters in one embodiment is shown by FIG. 4. As shown in fig. 4, there is provided a data processing apparatus 4 between database clusters, applied to a target database end, where the data processing apparatus 4 includes a receiving module 40, an obtaining module 41, and a pulling module 42, where:
a receiving module 40, configured to send a data pulling request to a service gateway, and receive a data source table structure returned by the service gateway;
the obtaining module 41 is configured to create a local temporary table according to the data source table structure, and obtain data of a corresponding type from the source database according to the data source type indicated in the data pull request;
and the pulling module 42 is used for pulling the corresponding data from the source database end to the local temporary table so as to complete the data processing.
The data processing device among the database clusters sends a data pulling request to the service gateway when being applied to the target database end and receives a data source end table structure returned by the service gateway; according to the data source end table structure, a local temporary table is created, and data of a corresponding type is obtained from a source database end according to the data source type indicated in the data pulling request; and pulling the corresponding data from the source database end to the local temporary table to finish data processing. The data processing device among the database clusters can realize data exchange and analysis across the database clusters.
In one embodiment, the receiving module 40 is specifically configured to:
In the case where the scene is an isomorphic scene: sending a data pulling request to a service gateway, wherein the data pulling request is used for indicating the service gateway to send a Structured Query Language (SQL) of a data source end table structure to a source database end and receiving the data source end table structure from the source database end; a data source end table structure is received from a service gateway.
In one embodiment, the receiving module 40 is further specifically configured to:
In the case where the scene is a heterogeneous scene: sending a data pulling request to a service gateway, wherein the data pulling request is used for indicating the service gateway to send a Structured Query Language (SQL) of a data source end table structure to a source database end and receiving the data source end table structure from the source database end; judging the type of a source database terminal according to the data pulling request, sending a Structured Query Language (SQL) of a data source terminal list structure to the source database terminal, formatting the heterogeneous database terminal into a format corresponding to the data source terminal list structure at the source database terminal, and receiving the data source terminal list structure from the source database terminal.
In one embodiment, the obtaining module 41 is specifically configured to:
in the case where the scene is an isomorphic scene:
Judging whether the data nodes of the target database end and the source database end can communicate or not;
If yes, directly acquiring a data file of a corresponding type from a source database terminal according to the data source type indicated in the data pulling request;
if not, the control service gateway executes a query statement according to the type of the source database terminal to acquire data, and formats the data into the data of the corresponding type.
In one embodiment, the obtaining module 41 is further specifically configured to:
in the case where the scene is a heterogeneous scene:
and the control service gateway executes a query statement according to the type of the source database end to acquire data, and formats the data into data of a corresponding type.
The modules in the data processing device between database clusters may be implemented in whole or in part by software, hardware, or a combination thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
In one embodiment, a computer device is provided, which may be a server, and the internal structure of which may be as shown in fig. 5. The computer device includes a processor, a memory, a network interface, and a transceiver connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The transceiver of the computer device is used for executing the operation of receiving data or transmitting data under the control of the processor. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device is used for storing data such as sample data. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a method of processing data between database clusters.
It will be appreciated by persons skilled in the art that the architecture shown in fig. 5 is merely a block diagram of some of the architecture relevant to the present inventive arrangements and is not limiting of the computer device to which the present inventive arrangements are applied, and in particular, a computer device may include more or less components than those shown, or may combine some of the components, or have a different arrangement of components.
In one embodiment, a computer device is provided, including a memory and a processor, where the memory stores a computer program, and the processor implements principles and specific procedures in each embodiment, and may be referred to in the foregoing embodiments for a description of an embodiment of a data processing method between database clusters, which is not described herein.
In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, where the principle and specific procedures in implementing embodiments when the computer program is executed by a processor may be referred to in the foregoing embodiments for a data processing method between database clusters in the foregoing embodiments, and are not repeated herein.
In one embodiment, a computer program product is provided, which includes a computer program, where the principle and specific procedures in implementing embodiments may be referred to in the foregoing embodiments of the data processing method between database clusters when the computer program is executed by a processor, and will not be described herein.
It should be noted that, the information related to the present application (including, but not limited to, information related to the data processing process between database clusters in the present application) is information or data fully authorized by each party.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (ReRAM), magneto-resistive random access Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (PHASE CHANGE Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in various forms such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The databases referred to in the embodiments provided herein may include at least one of a relational database and a non-relational database. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processor referred to in the embodiments provided in the present application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, or the like, but is not limited thereto.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The foregoing examples illustrate only a few embodiments of the application and are described in detail herein without thereby limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of the application should be assessed as that of the appended claims.
Claims (9)
1. A method for processing data among database clusters, which is applied to a target database end, the method comprising:
Sending a data pulling request to a service gateway, and receiving a data source end list structure returned by the service gateway;
creating a local temporary table according to the data source end table structure, and acquiring data of a corresponding type from a source database end according to the data source type indicated in the data pulling request;
And pulling corresponding data from the source database end to a local temporary table so as to finish data processing.
2. The method of claim 1, wherein the sending the data pull request to the service gateway and receiving the data source table structure returned by the service gateway comprise:
in the case where the scene is an isomorphic scene:
Sending a data pulling request to a service gateway, wherein the data pulling request is used for indicating the service gateway to send a Structured Query Language (SQL) of a data source end list structure to a source database end and receiving the data source end list structure from the source database end;
and receiving the data source end table structure from the service gateway.
3. The method of claim 1, wherein the sending the data pull request to the service gateway and receiving the data source table structure returned by the service gateway comprise:
in the case where the scene is a heterogeneous scene:
Sending a data pulling request to a service gateway, wherein the data pulling request is used for indicating the service gateway to send a Structured Query Language (SQL) of a data source end list structure to a source database end and receiving the data source end list structure from the source database end; judging the type of a source database terminal according to the data pulling request, sending a Structured Query Language (SQL) of a data source terminal list structure to the source database terminal, formatting heterogeneous database terminals into a format corresponding to the data source terminal list structure at the source database terminal, and receiving the data source terminal list structure from the source database terminal;
and receiving the data source end table structure from the service gateway.
4. The method according to claim 1, wherein the obtaining, from a source database side, the data of the corresponding type according to the data source type indicated in the data pull request includes:
in the case where the scene is an isomorphic scene:
Judging whether the data nodes of the target database end and the source database end can communicate or not;
If yes, directly acquiring a data file of a corresponding type from a source database terminal according to the data source type indicated in the data pulling request;
if not, the control service gateway executes a query statement according to the type of the source database terminal to acquire data, and formats the data into data of a corresponding type.
5. The method according to claim 1, wherein the obtaining, from a source database side, the data of the corresponding type according to the data source type indicated in the data pull request includes:
in the case where the scene is a heterogeneous scene:
and the control service gateway executes a query statement according to the type of the source database terminal to acquire data, and formats the data into data of a corresponding type.
6. A data processing apparatus between database clusters, applied to a target database end side, the apparatus comprising:
The receiving module is used for sending a data pulling request to the service gateway and receiving a data source end list structure returned by the service gateway;
The acquisition module is used for creating a local temporary table according to the data source end table structure and acquiring corresponding type data from a source database end according to the data source type indicated in the data pulling request;
And the pulling module is used for pulling the corresponding data from the source database end to the local temporary table so as to finish data processing.
7. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any one of claims 1 to 5 when the computer program is executed.
8. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 5.
9. A computer program product comprising a computer program, characterized in that the computer program, when being executed by a processor, implements the steps of the method according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410789311.4A CN118364033B (en) | 2024-06-19 | 2024-06-19 | Data processing method and device among database clusters and related equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410789311.4A CN118364033B (en) | 2024-06-19 | 2024-06-19 | Data processing method and device among database clusters and related equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN118364033A true CN118364033A (en) | 2024-07-19 |
| CN118364033B CN118364033B (en) | 2024-10-11 |
Family
ID=91885918
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410789311.4A Active CN118364033B (en) | 2024-06-19 | 2024-06-19 | Data processing method and device among database clusters and related equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118364033B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020114281A1 (en) * | 2001-02-22 | 2002-08-22 | Telefonaktiebolaget Lm Ericsson | Method and apparatus for performance improvement of multi-service networks |
| US20100082671A1 (en) * | 2008-09-26 | 2010-04-01 | International Business Machines Corporation | Joining Tables in Multiple Heterogeneous Distributed Databases |
| CN104484468A (en) * | 2014-12-31 | 2015-04-01 | 天津南大通用数据技术股份有限公司 | Database system based on Dblink and transparent gateway and construction using method of database system |
| CN115033639A (en) * | 2022-04-14 | 2022-09-09 | 中国农业银行股份有限公司 | Method and related device for generating relation graph for data sharing among clusters |
| CN115511501A (en) * | 2021-06-03 | 2022-12-23 | 腾讯云计算（北京）有限责任公司 | Data processing method, computer equipment and readable storage medium |
| CN117056305A (en) * | 2023-08-07 | 2023-11-14 | 中移动信息技术有限公司 | Construction method, model, database system and medium of multisource isomorphic database |
| CN117311898A (en) * | 2023-09-01 | 2023-12-29 | 广西电网有限责任公司 | Heterogeneous data acquisition method and system based on K8S cluster |
| CN117914787A (en) * | 2023-12-08 | 2024-04-19 | 福思(杭州)智能科技有限公司 | Data transmission method, device, electronic device and storage medium |
-
2024
- 2024-06-19 CN CN202410789311.4A patent/CN118364033B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020114281A1 (en) * | 2001-02-22 | 2002-08-22 | Telefonaktiebolaget Lm Ericsson | Method and apparatus for performance improvement of multi-service networks |
| US20100082671A1 (en) * | 2008-09-26 | 2010-04-01 | International Business Machines Corporation | Joining Tables in Multiple Heterogeneous Distributed Databases |
| CN104484468A (en) * | 2014-12-31 | 2015-04-01 | 天津南大通用数据技术股份有限公司 | Database system based on Dblink and transparent gateway and construction using method of database system |
| CN115511501A (en) * | 2021-06-03 | 2022-12-23 | 腾讯云计算（北京）有限责任公司 | Data processing method, computer equipment and readable storage medium |
| CN115033639A (en) * | 2022-04-14 | 2022-09-09 | 中国农业银行股份有限公司 | Method and related device for generating relation graph for data sharing among clusters |
| CN117056305A (en) * | 2023-08-07 | 2023-11-14 | 中移动信息技术有限公司 | Construction method, model, database system and medium of multisource isomorphic database |
| CN117311898A (en) * | 2023-09-01 | 2023-12-29 | 广西电网有限责任公司 | Heterogeneous data acquisition method and system based on K8S cluster |
| CN117914787A (en) * | 2023-12-08 | 2024-04-19 | 福思(杭州)智能科技有限公司 | Data transmission method, device, electronic device and storage medium |
Non-Patent Citations (3)
| Title |
|---|
| RASHA KASHEF等: "Homogeneous Vs. Heterogeneous Distributed Data Clustering: A Taxonomy", SPRINGER LINK, 21 December 2019 (2019-12-21), pages 51 - 56 * |
| 南大通用: "南大通用产品文档：GBase8a", pages 1 - 6, Retrieved from the Internet <URL:http://gdoc.gbase.cn/gbase-test-01/gbase-knowledge-text-10408.html> * |
| 苏乐: "事务中间件在高可用性数据库系统中的应用", 中国优秀硕士学位论文全文数据库, 15 August 2012 (2012-08-15) * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN118364033B (en) | 2024-10-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112000741B (en) | Internal and external network data exchange system, method, device, computer equipment and medium | |
| CN110032604A (en) | Data storage device, transfer device and data bank access method | |
| EP3470992B1 (en) | Efficient storage and utilization of a hierarchical data set | |
| CN111258978A (en) | Data storage method | |
| US20200401625A1 (en) | Graph processing system | |
| CN111818175B (en) | Enterprise service bus configuration file generation method, device, equipment and storage medium | |
| CN114218218A (en) | Data processing method, device and equipment based on data warehouse and storage medium | |
| KR20020050160A (en) | Object integrated management system | |
| CN113961643A (en) | Search engine updating method and device, equipment, medium and product thereof | |
| CN112970011B (en) | Pedigree in record query optimization | |
| CN114328981B (en) | Knowledge graph establishing and data acquiring method and device based on mode mapping | |
| WO2017107130A1 (en) | Data query method and database system | |
| CN115858322A (en) | Log data processing method and device and computer equipment | |
| CN112860412B (en) | Service data processing method and device, electronic equipment and storage medium | |
| CN118364033B (en) | Data processing method and device among database clusters and related equipment | |
| CN115858471A (en) | Service data change recording method, device, computer equipment and medium | |
| CN116010345A (en) | Method, device and equipment for realizing table service scheme of flow batch integrated data lake | |
| CN112685557B (en) | Visual information resource management method and device | |
| US20220254462A1 (en) | System and Method of Property Collection Management and Architecture | |
| CN112185494B (en) | Data storage method, device, computer equipment and storage medium | |
| CN116880927A (en) | Rule management method, device, computer equipment and storage medium | |
| CN116049200A (en) | Method, device, computer equipment and medium for updating analysis library data | |
| CN118331936A (en) | IFC file compression method and device, electronic equipment and storage medium | |
| CN118820323A (en) | Track data processing method, apparatus, computer device, readable storage medium, and program product | |
| CN116227461A (en) | Method, device, equipment, medium and product for generating report with custom format |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |