CN117493465A - Data circulation method and device, electronic equipment and storage medium - Google Patents
Data circulation method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN117493465A CN117493465A CN202311744339.8A CN202311744339A CN117493465A CN 117493465 A CN117493465 A CN 117493465A CN 202311744339 A CN202311744339 A CN 202311744339A CN 117493465 A CN117493465 A CN 117493465A
- Authority
- CN
- China
- Prior art keywords
- data
- configuration table
- slave
- metadata
- master
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 45
- 238000012546 transfer Methods 0.000 claims abstract description 38
- 238000004590 computer program Methods 0.000 claims description 16
- 238000004891 communication Methods 0.000 claims description 13
- 239000000284 extract Substances 0.000 claims description 13
- 238000013508 migration Methods 0.000 claims description 4
- 230000005012 migration Effects 0.000 claims description 4
- 101000822425 Arthrobacter sp. (strain KUJ 8602) Guanidinobutyrase Proteins 0.000 description 14
- 238000010586 diagram Methods 0.000 description 8
- 238000012545 processing Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 6
- 238000013075 data extraction Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 239000000047 product Substances 0.000 description 2
- 239000008186 active pharmaceutical agent Substances 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000035755 proliferation Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a data transfer method, a data transfer device, electronic equipment and a storage medium. The method specifically comprises the following steps: acquiring a preset metadata configuration table, a metadata field configuration table and a data flow requirement; determining a master end and a slave end of data to be transferred according to data transfer requirements; and the control master terminal performs data flow with the slave terminal according to the metadata configuration table and the metadata field configuration table. According to the technical scheme, the master end and the slave end of each data to be transferred are determined according to the data transfer requirement, and then the master end is controlled to transfer the data to the slave end according to the metadata configuration table and the metadata field configuration table which are configured in advance, so that the requirement of a user is met, the free transfer of big data is controlled, and the flexibility and the efficiency of data transfer are greatly improved.
Description
Technical Field
The present disclosure relates to the field of big data technologies, and in particular, to a data transfer method, a device, an electronic apparatus, and a storage medium.
Background
With the rapid development of internet technology, various industries have entered a big data age. With the consequent proliferation of data volumes and processing pressures for large data. Various types of large data platforms have been developed. Among them, especially, distributed relational databases emerge as if they were spring shoots after rain, and provide technical support for the construction and operation of data warehouses and data lakes for various industries. However, since each big data platform is not interconnected, there are still some problems in data circulation.
Currently, in various large data platforms, data flow between a non-relational database and a relational database is commonly used by extracting data in the non-relational database and pushing the data into the relational database. However, data cannot be extracted or pushed from other large data environment ends, so that the flexibility and efficiency of data flow are poor.
Disclosure of Invention
The application provides a data transfer method, a data transfer device, electronic equipment and a storage medium, so that the flexibility and the efficiency of data transfer are improved.
According to an aspect of the present application, there is provided a data circulation method, including:
acquiring a preset metadata configuration table, a metadata field configuration table and a data flow requirement;
determining a master end and a slave end of data to be transferred according to data transfer requirements;
and the control master terminal performs data flow with the slave terminal according to the metadata configuration table and the metadata field configuration table.
According to another aspect of the present application, there is provided a data circulation apparatus, the apparatus including:
the configuration table acquisition module is used for acquiring a preset metadata configuration table, a metadata field configuration table and a data flow requirement;
the master-slave end determining module is used for determining a master end and a slave end of data to be circulated according to the data circulation requirement;
and the data flow module is used for controlling the master terminal to conduct data flow with the slave terminal according to the metadata configuration table and the metadata field configuration table.
According to another aspect of the present application, there is provided an electronic device including:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,
the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the data streaming method described in any of the embodiments of the present application.
According to another aspect of the present application, there is provided a computer readable storage medium storing computer instructions for causing a processor to execute a data streaming method according to any of the embodiments of the present application.
According to the technical scheme, the master end and the slave end of each data to be transferred are determined according to the data transfer requirement, and then the master end is controlled to transfer the data to the slave end according to the metadata configuration table and the metadata field configuration table which are configured in advance, so that the requirement of a user is met, the free transfer of big data is controlled, and the flexibility and the efficiency of data transfer are greatly improved.
It should be understood that the description of this section is not intended to identify key or critical features of the embodiments of the application or to delineate the scope of the application. Other features of the present application will become apparent from the description that follows.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a flowchart of a data circulation method according to a first embodiment of the present application;
fig. 2A is a schematic diagram of data flow with Hadoop as a main end according to a second embodiment of the present application;
fig. 2B is a data flow schematic diagram of GBase as a primary end according to a second embodiment of the present application;
FIG. 2C is a diagram illustrating the data flow of a multi-type database adapted according to a second embodiment of the present application;
fig. 3 is a schematic structural diagram of a data circulation device according to a third embodiment of the present application;
fig. 4 is a schematic structural diagram of an electronic device implementing a data circulation method according to an embodiment of the present application.
Detailed Description
In order to make the present application solution better understood by those skilled in the art, the following description will be made in detail and with reference to the accompanying drawings in the embodiments of the present application, it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, shall fall within the scope of the present application.
It should be noted that the terms "first," "second," and the like in the description and claims of the present application and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that embodiments of the present application described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Example 1
Fig. 1 is a flowchart of a data flow method provided in an embodiment of the present application, where the embodiment is applicable to a case of performing data flow between a distributed relational database and non-relational data, and the method may be performed by a data flow device, where the data flow device may be implemented in a form of hardware and/or software, and the data flow device may be configured in an electronic device. As shown in fig. 1, the method includes:
s110, acquiring a preset metadata configuration table, a metadata field configuration table and a data flow requirement.
The metadata configuration table may be used to store metadata information of the data stream, where the metadata information may include, but is not limited to, source data table names, group names, database node addresses, data file storage paths, database connection addresses, target data table names, and the like. The metadata field configuration table may be used to store database fields corresponding to metadata, and may include, for example, but not limited to, a data table name, a field name, an identifier, a data type, a primary key, and the like. The data flow requirement can be a request of a user of a large data platform for data flow between different databases, and the data flow comprises data extraction and data pushing. The data flow requirement may include, but is not limited to, a group name and a data flow direction of the data to be circulated. The group name may be the name of the data group in the database used to locate the data, and the direction of data flow may be where the data needs to be streamed to.
Because the metadata configuration table and the metadata field configuration table are preset, the metadata field configuration table can be directly acquired as the data flow requirement. Of course, the metadata configuration table and the metadata field configuration table may be set by related staff according to actual situations, which is not limited in the embodiment of the present application.
S120, determining a master end and a slave end of data to be transferred according to the data transfer requirement.
The data to be transferred can be various databases which need to be transferred, and correspondingly, the databases of the data to be transferred can be used for storing the data to be transferred and can also be used for transferring the data, and at least two databases of the data to be transferred exist. It will be appreciated that in the process of data flowing from one end to the other, the data needs to be extracted from the master end to the slave end, and the data needs to be pushed from the master end to the slave end. Since the data flow direction is included in the data flow requirement, it can be determined where the data is extracted from or pushed to, so as to determine the master and slave of all the data to be circulated.
S130, the control master terminal performs data flow with the slave terminal according to the metadata configuration table and the metadata field configuration table.
And the master terminal searches the content needed to be used in the data stream through a preset metadata configuration table and a metadata field configuration table, and sequentially controls the data stream between the master terminal and the slave terminal according to the credentials. In an alternative embodiment, the master end performs data circulation with the slave end according to the metadata configuration table and the metadata field configuration table, and may include: and the master terminal extracts the slave terminal data from the slave terminal according to the metadata configuration table and the metadata field configuration table, and pushes the master terminal data to the slave terminal so as to carry out data circulation.
It can be understood that the metadata configuration table can be queried to obtain information such as a source data table name, a data file storage path, a database connection address and the like, and related information such as a database driver name, a port number, a user name, a field name and the like can be queried to obtain from the metadata field configuration table. According to the queried information, the master terminal is controlled to extract slave terminal data from the slave terminal to the master terminal according to the address, the port and the like obtained from the information, and then the master terminal data is pushed to the slave terminal according to the address, the port and the like, so that the data flow meeting the data flow requirement of a user is achieved.
Optionally, the database of the data to be circulated includes a distributed non-relational database and a distributed relational database.
The distributed databases may be divided into non-relational databases and relational databases, and compared with the data flow between relational databases, the prior art has few solutions for providing data flow between non-relational databases and relational databases. The present embodiment will be described taking as an example a data flow between a non-relational database and a relational database. The distributed non-relational database can adopt Hadoop, and the distributed relational database can adopt GBase.
In an alternative embodiment, if the master end is a distributed non-relational database and the slave end is a distributed relational database, the master end extracts the slave end data from the slave end according to the metadata configuration table and the metadata field configuration table and pushes the master end data to the slave end, which may include: and the distributed non-relational database extracts the slave-end data from the distributed relational database through a preset data migration component according to the metadata configuration table and the metadata field configuration table, and pushes the master-end data.
The data migration component may employ a Sqoop component, among other things. The component extracts data from the distributed relational database and pushes data in the distributed non-relational database to the distributed relational data. Taking Hadoop and GBase as examples, when the Hadoop is taken as a main end, data is extracted and pushed through the Sqoop component. Sqoop is used as a component of the Hadoop cluster and is responsible for transmitting data between the Hadoop and the relational databases, so that the data in one relational database can be extracted into the Hadoop HDFS (Hadoop Distributed File System, distributed file system) or the data of the HDFS can be pushed into the relational database. By querying information in a metadata configuration table and a metadata table field configuration table, data flow is carried out between the metadata configuration table and the slave GBase through Sqoop by taking Hadoop as a master end.
In another alternative embodiment, if the master end is a distributed relational database and the slave end is a distributed non-relational database, the master end extracts the slave end data from the slave end according to the metadata configuration table and the metadata field configuration table, and pushes the master end data to the slave end, which may include: and the distributed relational database extracts the slave-end data from the distributed non-relational database through a preset communication protocol according to the metadata configuration table and the metadata field configuration table, and pushes the master-end data.
Alternatively, the preset communication protocol may employ WebHDFS protocol, a protocol defining public HTTP REST API, and the API (Application ProgrammingInterface ) allows clients to access Hadoop Distributed File Systems (HDFS) via the Web. By querying information in a metadata configuration table and a metadata table field configuration table, data flow is carried out between the metadata configuration table and a slave Hadoop by using GBase as a master through WebHDFS.
In the embodiments, the distributed non-relational database and the non-relational database are provided as the data circulation schemes when the master end and the slave end are respectively used, so that a practical and effective solution is provided for data communication between the non-relational database and the relational database, the flexibility of data circulation among various databases is further improved, the interconnection and intercommunication of data are more convenient, and the improvement of the efficiency of data circulation is facilitated.
In addition, it is necessary to supplement that in the process of performing data transfer between the distributed non-relational database and the distributed relational database, multiple parties are not allowed to perform writing operation on the same table of the same database at the same time, and are not allowed to perform writing operation while performing reading operation on the same table of the same database, so as to avoid causing mutual coverage, inconsistency or data error of data.
According to the technical scheme, the master end and the slave end of each data to be transferred are determined according to the data transfer requirement, and then the master end is controlled to transfer the data to the slave end according to the metadata configuration table and the metadata field configuration table which are configured in advance, so that the requirement of a user is met, the free transfer of big data is controlled, and the flexibility and the efficiency of data transfer are greatly improved.
Example two
The present embodiment is a preferred embodiment provided on the basis of the foregoing embodiments. As shown in fig. 2A, 2B and 2C, the following is specific:
firstly, setting a metadata configuration component, and configuring a data table name, a Hadoop database connection address, a data file storage path, a distributed relational database connection address and the like. The metadata configuration table is defined and used for storing metadata information of data stream, and specifically comprises fields such as a source data table name, a group name, a Hadoop NameNode address, a data file storage path, a distributed relational database connection address, a target data table name and the like, as shown in table 1.
TABLE 1
In addition, a "metadata table field configuration table" is also required to be preset for storing the data fields contained in each data table, as shown in table 2.
TABLE 2
Then, a 'Hadoop end data scheduling' component is set, and data is controlled to freely flow between the Hadoop and the distributed relational database at the Hadoop end. When Hadoop is used as a main end, as shown in fig. 2A, data is extracted and pushed through the Sqoop component. The Sqoop is used as a component of the Hadoop cluster and is responsible for transmitting data between the Hadoop and the relational databases, so that the data in one relational database can be extracted into the HDFS of the Hadoop, and the data of the HDFS can be pushed into the relational database. According to the group name and the data flow direction (data extraction or pushing) input by a user, inquiring a metadata configuration table to obtain information such as a source data table name, a data file storage path, a distributed relational database connection address and the like, and completing extraction of data from GBase to Hadoop or pushing of data from Hadoop to GBase at a Hadoop end through Sqoop. Common parameters of the Sqoop command include GBase connection address, port number, GBase driver name, user name, password, source data table name, destination data table name, field name (query "metadata table field configuration table" get), query conditions, and the like.
Then, a distributed relational database end data scheduling component is set, and data is controlled to freely flow between Hadoop and the distributed relational database at the distributed relational database end. With GBase as the master, as shown in FIG. 2B, data is extracted and pushed according to the WebHDFS protocol. The WebHDFS protocol is a protocol defining public HTTP REST API that allows clients to access Hadoop Distributed File Systems (HDFS) via the Web. The WebHDFS protocol provides a set of functions that enable users to directly process files and folders stored in Hadoop via REST APIs, performing reads, writes, uploads and downloads of files. According to the group name and the data flow direction (data extraction or pushing) input by a user, inquiring a metadata configuration table to obtain information such as a source data table name, a data file storage path, a Hadoop NameNode address and the like, and completing extraction of data from Hadoop to GBase or pushing of data from GBase to Hadoop at a GBase end according to a WebHDFS protocol. Common parameters of the WebHDFS protocol include source data table name, nameNode IP address, port number, data file storage path, destination data table name, user name, field delimiter, row delimiter, etc. Of course, based on the above two cases, the case of data of the multi-type database is mutually circulated as shown in fig. 2C.
Finally, a data scheduling master control component is arranged to control the Hadoop end data scheduling and the distributed relational database end data scheduling, so that the conditions that the same data table is simultaneously extracted and pushed are avoided. The data scheduling master control component coordinates and controls all data scheduling of the Hadoop end and the GBase end, does not allow the same data table of the same database to be simultaneously written, does not allow the same data table of the same database to be simultaneously read and written, avoids causing data to be mutually covered, inconsistent or data errors, and only allows the same data table of the same database to be simultaneously read. And writing operation or reading and writing operation of the same data table of the same database is put into a queue according to the sequence of the request submitting time, and the previous request is processed and then the next request is processed.
The traditional mode only provides a method for carrying out data extraction by taking the Hadoop end as a main end, and does not relate to pushing Hadoop data processing results to other big data environments, or extracting Hadoop data or pushing data to Hadoop from other big data environments. The invention realizes the data transfer method and the device of the Hadoop and the distributed relational database based on double-end control, and according to user configuration and operation scheduling, the free transfer of data between the Hadoop and the distributed relational database can be controlled at both the Hadoop end and the distributed relational database end, the large data transfer aspect is more convenient and efficient, and the flexibility and the efficiency of data transfer between various databases are greatly improved.
Example III
Fig. 3 is a schematic structural diagram of a data circulation device according to a third embodiment of the present application. As shown in fig. 3, the apparatus 300 includes:
a configuration table obtaining module 310, configured to obtain a metadata configuration table and a metadata field configuration table that are set in advance, and a data stream requirement;
a master-slave end determining module 320, configured to determine a master end and a slave end of data to be transferred according to a data transfer requirement;
the data flow module 330 is configured to control the master terminal to perform data flow with the slave terminal according to the metadata configuration table and the metadata field configuration table.
According to the technical scheme, the master end and the slave end of each data to be transferred are determined according to the data transfer requirement, and then the master end is controlled to transfer the data to the slave end according to the metadata configuration table and the metadata field configuration table which are configured in advance, so that the requirement of a user is met, the free transfer of big data is controlled, and the flexibility and the efficiency of data transfer are greatly improved.
In an alternative embodiment, the data flow module 330 may be specifically configured to:
and the master terminal extracts the slave terminal data from the slave terminal according to the metadata configuration table and the metadata field configuration table, and pushes the master terminal data to the slave terminal so as to carry out data circulation.
In an alternative embodiment, the database of data to be circulated includes a distributed non-relational database and a distributed relational database.
In an alternative embodiment, if the master is a distributed non-relational database and the slave is a distributed relational database, the data flow module 330 may be specifically configured to:
and the distributed non-relational database extracts the slave-end data from the distributed relational database through a preset data migration component according to the metadata configuration table and the metadata field configuration table, and pushes the master-end data.
In an alternative embodiment, if the master is a distributed relational database and the slave is a distributed non-relational database, the data flow module 330 may be specifically configured to:
and the distributed relational database extracts the slave-end data from the distributed non-relational database through a preset communication protocol according to the metadata configuration table and the metadata field configuration table, and pushes the master-end data.
In an alternative embodiment, the data transfer requirement includes a group name and a data transfer direction of the data to be transferred.
The data transfer device provided by the embodiment of the application can execute the data transfer method provided by any embodiment of the application, and has the corresponding functional modules and beneficial effects of executing each data transfer method.
Example IV
Fig. 4 shows a schematic diagram of the structure of an electronic device 10 that may be used to implement embodiments of the present application. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Electronic equipment may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the application described and/or claimed herein.
As shown in fig. 4, the electronic device 10 includes at least one processor 11, and a memory, such as a Read Only Memory (ROM) 12, a Random Access Memory (RAM) 13, etc., communicatively connected to the at least one processor 11, in which the memory stores a computer program executable by the at least one processor, and the processor 11 may perform various appropriate actions and processes according to the computer program stored in the Read Only Memory (ROM) 12 or the computer program loaded from the storage unit 18 into the Random Access Memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 may also be stored. The processor 11, the ROM 12 and the RAM 13 are connected to each other via a bus 14. An input/output (I/O) interface 15 is also connected to bus 14.
Various components in the electronic device 10 are connected to the I/O interface 15, including: an input unit 16 such as a keyboard, a mouse, etc.; an output unit 17 such as various types of displays, speakers, and the like; a storage unit 18 such as a magnetic disk, an optical disk, or the like; and a communication unit 19 such as a network card, modem, wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.
The processor 11 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, digital Signal Processors (DSPs), and any suitable processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as the data flow method.
In some embodiments, the data flow method may be implemented as a computer program tangibly embodied on a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 10 via the ROM 12 and/or the communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the data flow method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the data flow method in any other suitable way (e.g., by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.
A computer program for carrying out the methods of the present application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be implemented. The computer program may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of this application, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which a user can provide input to the electronic device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), blockchain networks, and the internet.
The computing system may include clients and servers. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical hosts and VPS service are overcome.
It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps described in the present application may be performed in parallel, sequentially, or in a different order, so long as the desired results of the technical solutions of the present application are achieved, and the present application is not limited herein.
The above embodiments do not limit the scope of the application. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application are intended to be included within the scope of the present application.
Claims (10)
1. A method of data streaming, the method comprising:
acquiring a preset metadata configuration table, a metadata field configuration table and a data flow requirement;
determining a master end and a slave end of data to be transferred according to the data transfer requirement;
and controlling the master terminal to conduct data flow with the slave terminal according to the metadata configuration table and the metadata field configuration table.
2. The method of claim 1, wherein the master performs data streaming with the slave according to the metadata configuration table and the metadata field configuration table, comprising:
and the master terminal extracts slave terminal data from the slave terminal according to the metadata configuration table and the metadata field configuration table, and pushes the master terminal data to the slave terminal so as to carry out data circulation.
3. The method of claim 2, wherein the database of data to be streamed comprises a distributed non-relational database and a distributed relational database.
4. The method of claim 3, wherein if the master is a distributed non-relational database and the slave is a distributed relational database, the master extracting slave data from the slave and pushing master data to the slave according to the metadata configuration table and the metadata field configuration table, comprising:
and the distributed non-relational database extracts the slave-end data from the distributed relational database through a preset data migration component according to the metadata configuration table and the metadata field configuration table, and pushes the master-end data.
5. The method of claim 3, wherein if the master is a distributed relational database and the slave is a distributed non-relational database, the master extracting slave data from the slave and pushing master data to the slave according to the metadata configuration table and the metadata field configuration table, comprising:
and the distributed relational database extracts the slave-end data from the distributed non-relational database through a preset communication protocol according to the metadata configuration table and the metadata field configuration table, and pushes the master-end data.
6. The method according to any of claims 1-5, wherein the data flow requirements include a group name and a data flow direction of the data to be circulated.
7. A data streaming apparatus, the apparatus comprising:
the configuration table acquisition module is used for acquiring a preset metadata configuration table, a metadata field configuration table and a data flow requirement;
the master-slave end determining module is used for determining a master end and a slave end of data to be circulated according to the data circulation requirement;
and the data flow module is used for controlling the master terminal to conduct data flow with the slave terminal according to the metadata configuration table and the metadata field configuration table.
8. The apparatus of claim 7, wherein the data flow module is specifically configured to:
and the master terminal extracts slave terminal data from the slave terminal according to the metadata configuration table and the metadata field configuration table, and pushes the master terminal data to the slave terminal so as to carry out data circulation.
9. An electronic device, the electronic device comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,
the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the data flow method of any one of claims 1-6.
10. A computer readable storage medium storing computer instructions for causing a processor to perform the data transfer method of any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311744339.8A CN117493465A (en) | 2023-12-18 | 2023-12-18 | Data circulation method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311744339.8A CN117493465A (en) | 2023-12-18 | 2023-12-18 | Data circulation method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117493465A true CN117493465A (en) | 2024-02-02 |
Family
ID=89669259
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311744339.8A Pending CN117493465A (en) | 2023-12-18 | 2023-12-18 | Data circulation method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117493465A (en) |
-
2023
- 2023-12-18 CN CN202311744339.8A patent/CN117493465A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20200004464A1 (en) | Method and apparatus for storing data | |
| CN111723161A (en) | Data processing method, device and equipment | |
| CN112528067A (en) | Graph database storage method, graph database reading method, graph database storage device, graph database reading device and graph database reading equipment | |
| US20190012596A1 (en) | Data processing system, method, and device | |
| CN111814020A (en) | Data acquisition method and device | |
| CN113541987A (en) | Method and device for updating configuration data | |
| CN111444148A (en) | Data transmission method and device based on MapReduce | |
| CN117493465A (en) | Data circulation method and device, electronic equipment and storage medium | |
| CN116108094A (en) | Data integration method and device, electronic equipment and storage medium | |
| CN114070889B (en) | Configuration method, traffic forwarding device, storage medium, and program product | |
| CN115905322A (en) | Service processing method and device, electronic equipment and storage medium | |
| CN113691403B (en) | Topology node configuration method, related device and computer program product | |
| CN112860796B (en) | Method, apparatus, device and storage medium for synchronizing data | |
| US10114864B1 (en) | List element query support and processing | |
| CN113556370A (en) | Service calling method and device | |
| CN115118654B (en) | Data forwarding method, system, device and program product under virtual network | |
| CN114650222B (en) | Parameter configuration method, device, electronic equipment and storage medium | |
| CN111949500B (en) | Resource matching method, device, electronic equipment and readable storage medium | |
| CN114553894B (en) | Data synchronization method, device, system and storage medium | |
| CN112311833B (en) | Data updating method and device | |
| CN111054083B (en) | Method and system for creating original roles in social network, server and medium | |
| WO2022141655A1 (en) | Method for distributed-type computation of simrank single source node similarity measure, and apparatus | |
| WO2023143061A1 (en) | Data access method and data access system thereof | |
| CN116561106B (en) | Configuration item data management method and system | |
| CN113434508B (en) | Method and apparatus for storing information |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |