CN111814435A - Database data transmission method, device, equipment and readable storage medium - Google Patents
Database data transmission method, device, equipment and readable storage medium Download PDFInfo
- Publication number
- CN111814435A CN111814435A CN202010710074.XA CN202010710074A CN111814435A CN 111814435 A CN111814435 A CN 111814435A CN 202010710074 A CN202010710074 A CN 202010710074A CN 111814435 A CN111814435 A CN 111814435A
- Authority
- CN
- China
- Prior art keywords
- task
- data transmission
- information
- file
- database data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000005540 biological transmission Effects 0.000 title claims abstract description 115
- 238000000034 method Methods 0.000 title claims abstract description 70
- 230000006870 function Effects 0.000 claims description 40
- 238000006243 chemical reaction Methods 0.000 claims description 31
- 238000012545 processing Methods 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims description 9
- 238000004364 calculation method Methods 0.000 description 16
- 238000012546 transfer Methods 0.000 description 8
- 238000004891 communication Methods 0.000 description 7
- 230000008676 import Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 230000005236 sound signal Effects 0.000 description 4
- 238000012360 testing method Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 239000008186 active pharmaceutical agent Substances 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/12—Use of codes for handling textual entities
- G06F40/151—Transformation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a database data transmission method, a device, equipment and a computer readable storage medium, wherein the method comprises the following steps: acquiring a first task file for executing a database data transmission task; converting the first task file into a second task file matched with the memory computing system; sending the second task file to the memory computing system so that the memory computing system can execute the database data transmission task by using the second task file; after the first task file is converted into the second task file, the second task file is sent to the memory computing system, so that the memory computing system can execute a database data transmission task by using the second task file. By adopting the memory computing system with higher computing efficiency, the data transmission speed between the databases can be improved, and the time required by data transmission is reduced.
Description
Technical Field
The present invention relates to the field of data transmission technologies, and in particular, to a database data transmission method, a database data transmission device, a database data transmission apparatus, and a computer-readable storage medium.
Background
In practical applications, there are situations where data transmission or transfer between different kinds of databases is required, such as transferring data from a relational database to a distributed file database, or transferring data from a distributed file database to a relational database. The related art generally uses a dedicated data transmission tool to export data from a designated database and import the exported data into another database, and these tools generally use a calculation method of the data transmission tool to perform calculation, for example, a MapReduce calculation method. Because the calculation methods have a slow operation speed, the data transmission speed is slow, and the time required by data transmission is long.
Therefore, how to solve the problems of slow data transmission speed and long data transmission time in the related art is a technical problem to be solved by those skilled in the art.
Disclosure of Invention
In view of the above, an object of the present invention is to provide a database data transmission method, a database data transmission apparatus, a database data transmission device, and a computer readable storage medium, which solve the problems of slow data transmission speed and long data transmission time in the related art.
In order to solve the technical problem, the invention provides a database data transmission method, which comprises the following steps:
acquiring a first task file for executing a database data transmission task;
converting the first task file into a second task file matched with a memory computing system;
and sending the second task file to the memory computing system so that the memory computing system executes the database data transmission task by using the second task file.
Optionally, the converting the first task file into a second task file matched with a memory computing system includes:
acquiring first input information, first output information and first function information from the first task file;
converting the first input information, the first output information and the first function information according to a format conversion relation to obtain second input information, second output information and second function information which are matched with the memory computing system;
and generating the second task file by using the second input information, the second output information and the second function information.
Optionally, the method further comprises:
acquiring first information and second information matched with the memory computing system;
and establishing a corresponding relation between the first information and the second information to obtain the format conversion relation.
Optionally, the obtaining a first task file for executing a database data transmission task includes:
acquiring task information and a command script file;
and generating the first task file according to the task information by using the command script file.
Optionally, the method further comprises:
generating a plurality of commands according to the command rule;
and generating the command script file by using the command.
Optionally, the obtaining a first task file for executing a database data transmission task includes:
and acquiring the first task file sent by the target equipment.
Optionally, the method further comprises:
and feeding back task information to the target equipment.
The invention also provides a database data transmission device, comprising:
the system comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring a first task file for executing a database data transmission task;
the conversion module is used for converting the first task file into a second task file matched with the memory computing system;
and the sending module is used for sending the second task file to the memory computing system so that the memory computing system can execute the database data transmission task by using the second task file.
The invention also provides a database data transmission device, comprising a memory and a processor, wherein:
the memory is used for storing a computer program;
the processor is used for executing the computer program to realize the database data transmission method.
The present invention also provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the database data transmission method described above.
The database data transmission method provided by the invention comprises the steps of acquiring a first task file for executing a database data transmission task; converting the first task file into a second task file matched with the memory computing system; and sending the second task file to the memory computing system so that the memory computing system executes the database data transmission task by using the second task file.
Therefore, when the first task file for performing the database data transmission task is obtained, the method is not directly executed by using the original calculation mode of the data transmission tool, but is converted into the second task file matched with the memory calculation system. The memory computing system can realize high-speed computing, and has higher computing efficiency than the original computing mode of a data transmission tool. And after the first task file is converted into a second task file, the second task file is sent to the memory computing system, so that the memory computing system executes the database data transmission task by using the second task file. By adopting the memory computing system with higher computing efficiency, the data transmission speed between databases can be improved, the time required by data transmission is reduced, and the problems of lower data transmission speed and longer data transmission time in the related technology are solved.
In addition, the invention also provides a database data transmission device, a database data transmission device and a computer readable storage medium, and the beneficial effects are also achieved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
Fig. 1 is a flowchart of a database data transmission method according to an embodiment of the present invention;
fig. 2 is a schematic structural diagram of a database data transmission apparatus according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of a database data transmission device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In a possible implementation manner, please refer to fig. 1, where fig. 1 is a flowchart of a database data transmission method according to an embodiment of the present invention. The method comprises the following steps:
s101: a first task file for performing a database data transfer task is obtained.
The first task file is a file for executing a database data transfer task, which may be specifically generated by the data transfer tool. The file format of the first task file is related to the data transmission tool, and the specific content of the first task file is related to the database of the exported data and the database of the imported data. The specific form of the data transmission tool can be a script, an executable file and the like. Different data transmission tools can transmit data among different types of databases. For example, if the data transfer tool is a Sqoop tool, it may perform data transfer between the relational database and the Hadoop distributed database. Data may be imported from a Relational Database management System (RDBMS), such as MySQL or Oracle, into a Hadoop Distributed File System (HDFS) or exported from the Hadoop Distributed File System (HDFS) to a Relational Database, such as MySQL, using the Sqoop tool. The Sqoop tool adopts a MapReduce computing mode (a computing model, a frame and a platform facing big data parallel processing) of Hadoop to extract and load data.
In one embodiment, the first task file may be sent by other terminals or devices, and in this case, the step S101 may include:
s1011: and acquiring a first task file sent by the target equipment.
The target device may specifically be a server or a computer, which may control data transmission between the databases by sending the first task file. The target device may be communicatively connected to a device that performs all or part of the steps in this embodiment, and the specific connection may be a wired connection or a wireless connection.
In another embodiment, the first task file may be generated locally, and in this case, the step S101 may include:
s1012: and acquiring task information and a command script file.
S1013: and generating a first task file according to the task information by using the command script file.
The task information is used for specifically setting the database data transmission task, such as specifying which database (e.g., a target database) data is exported from, which database data is imported to (e.g., a specified database), which data is exported from the database of the exported data, an export format and an import format of the data, and the like. The task information can be manually input by a user, or an information file can be read from a preset path, and the task information is analyzed from the file.
The command script file is a data transmission tool configured and used in a script form, such as a Sqoop tool in a script form. The command script file may be written in advance so that the first task file is generated in time after the task information is acquired. The first task file may be generated according to the task information by using the command script file, and the specific generation method is not limited in this embodiment, and related technologies may be referred to.
Further, the generation process of the command script file may include:
step 11: a plurality of commands are generated according to the command rules.
Step 12: a command script file is generated using the commands.
The command rule is a writing rule of the data transmission tool, that is, how to write a command capable of normally running, for example, the naming rule is a Sqoop command rule, and the obtained command is a Sqoop command. The command may be a function that needs to be called and executed in the data transmission process, and the specific content of the command is not limited, and may include, for example, an export function, a format conversion function, an import function, and the like. After the command is obtained, a command script file is generated by using the command script file, for example, the command can be saved as a shell script file, and a command script file in a shell file format is obtained.
S102: and converting the first task file into a second task file matched with the memory computing system.
The first task file is generated by the corresponding data transfer tool and is therefore computationally related to the data transfer tool. The original calculation mode of the data transmission tool is not limited, for example, the calculation mode of the Sqoop tool Hadoop MapReduce is used for extracting and loading data. Because the calculation mode of the data transmission tool has low calculation speed and low efficiency, in order to improve the efficiency, a memory calculation system can be used for calculation. The memory computing system may be a Spark computing system, or may be another computing system, which is not limited in this embodiment. The Spark computing system is a fast, general-purpose memory computing system. It provides high level APIs (interfaces) in Java, Scala, Python, and R languages, as well as an optimization engine that supports regular graphs. It also supports a rich set of higher-level tools, including sparkSQL, GraphX, etc. sparkSQL can be used for SQL and structured data processing, and GraphX can be used for graphics processing and spare Streaming (an easily extensible, high-throughput, high-fault-tolerant Streaming data processing system).
After the first task file is obtained, it may be converted into a second task file that matches the memory computing system. Specifically, the information in the first task file may be sequentially converted according to a format conversion relationship; or the corresponding initial second task file may be determined according to information (e.g., file format, corresponding data transmission tool, etc.) of the first task file, and after the task information in the first task file is extracted, the second task file is generated by using the task information and the initial second task file.
In one embodiment, in order to accurately convert the first task file, the situation that the generated second task file has errors or does not correspond to the first task file due to the fact that a template such as an initial second task file is not matched with the first task file is prevented. The S102 step may include:
s1021: first input information, first output information and first function information are acquired from a first task file.
S1022: and converting the first input information, the first output information and the first function information according to the format conversion relation to obtain second input information, second output information and second function information which are matched with the memory computing system.
S1023: and generating a second task file by using the second input information, the second output information and the second function information.
The first input information may also be referred to as DataInput, and is used to set an import format of data, where the import format of data is a format in which data is read and stored, for example, text file data is read and stored as a character string. The specific form of the first input information is a format matched by a data transmission tool, and may be an array, a vector, a matrix or other possible ways, for example. Similar to the first input information, the first output information may be referred to as DataOutput for setting a derived format of the data. The export format is what format of data is exported. The first function information may also be referred to as a map method, and represents a function employed in the calculation stage, for example, a function of reading a text file of the HDFS.
The format conversion relationship specifies the corresponding relationship of each item of data between the first task file and the second task file, and after the first input information, the first output information and the first function information are acquired from the first task file, the format conversion relationship is used for converting the first input information, the first output information and the first function information to acquire corresponding information. Specifically, the format conversion relationship may be traversed by using information such as the first input information, and when a matching correspondence relationship is found, another information that establishes a correspondence relationship with the information such as the first input information may be determined as information such as the second input information. The second input information, the second output information and the second function information obtained after conversion are substantially the same as the first input information, the first output information and the first function information before conversion, and the difference is that the matching objects are different, the information before conversion is matched with a data transmission tool, and the information after conversion is matched with a memory computing system. After the second input information, the second output information, and the second function information are obtained, a second task file matched with the memory computing system is generated by using the second input information, the second output information, and the second function information. The second task file may be in a jar packet format, or may be in other possible data formats.
For example, the first input information is Sqoop input information, the first output information is Sqoop output information, the first function information is map information, the second input information is Spark input information, the second output information is Spark output information, and the second function information is tansform information. When the first task file is used for exporting the text file in the HDFS to the relational database, the format conversion relationship is utilized to convert the Sqoop input information and the Sqoop output information into Spark input information and Spark output information which are matched with the memory computing system. The Sqoop input information and Spark input information are used for explaining that the format of reading and storing data is in a text format, and the Sqoop output information and Spark output information are used for explaining that the format of exporting data is in a format of corresponding relation database table fields. Since the format of the exported data is a text file format, the map information is a function for reading the text file, and the tansform information is a textfile method in Sparkcontext.
In one embodiment, before converting the first task file, the method may further include:
step 21: and acquiring the first information and second information matched with the memory computing system.
Step 22: and establishing a corresponding relation between the first information and the second information to obtain a format conversion relation.
The first information comprises first input information, first output information, first function information and other information, and the second information comprises second input information, second output information, second function information and other information. The two information are substantially the same, that is, the two information have different forms, but the functions in the corresponding calculation modes are the same, so that the corresponding relationship between the first information and the first information is established, and the format conversion relationship can be obtained. Specifically, it may be determined that the first information and the second information are substantially the same according to an instruction input by a user, or the first data and the second data may be tested by using test data, and whether the first information and the second information are substantially the same may be determined according to a test result, for example, when the same test data obtains the same test result, the first information and the second information are considered to be substantially the same.
S103: and sending the second task file to the memory computing system so that the memory computing system executes the database data transmission task by using the second task file.
And sending the second task file to a memory computing system after the second task file is obtained. And executing the second task file by the memory computing system after acquiring the second task file, namely executing the database data transmission task. In one possible implementation, if the first task file is sent by the target device, the task information may be fed back to the target device, so that the target device knows the task execution process in time.
By applying the database data transmission method provided by the embodiment of the invention, when the first task file for performing the database data transmission task is obtained, the first task file is directly executed without using the original calculation mode of the data transmission tool, and is converted into the second task file matched with the memory calculation system. The memory computing system can realize high-speed computing, and has higher computing efficiency than the original computing mode of a data transmission tool. And after the first task file is converted into a second task file, the second task file is sent to the memory computing system, so that the memory computing system executes the database data transmission task by using the second task file. By adopting the memory computing system with higher computing efficiency, the data transmission speed between databases can be improved, the time required by data transmission is reduced, and the problems of lower data transmission speed and longer data transmission time in the related technology are solved.
Based on the foregoing embodiments, in a possible implementation manner, the data transmission tool is a Sqoop tool, so that the conversion of the first task file may be performed by using a Sqoop job parser and a Sqoop job converter, and the second task file may be sent to the memory computing system by using a memory computing system job submitter.
After the Sqoop job parser obtains the Sqoop task job (first task file) generated by the user in the command writing mode supported by the Sqoop component, the Sqoop task job parser parses the Sqoop task job, obtains the input (first input information), the output (first output information) and the map method (first function information) of the Sqoop job by parsing, determines the input (second input information), the output (second output information) and the conversion method (TANSOFRM, second function information) supported by the corresponding memory computing system, and sends the input (second input information), the output (second output information) and the conversion method (TANSOFRM, second function information) to the Sqoop job converter. After the Sqoop job converter acquires the input and output of the Sqoop job and the corresponding relation between the map and the input and output and conversion methods supported by the memory computing system, the Sqoop job converter generates jobs (second task files) of the input and output and conversion methods and the like supported by the memory computing system according to the corresponding rules, and then sends the jobs to a memory computing system job submitter. And the memory computing system job submitter submits the job converted by the Sqoop job converter to the memory computing system cluster and delivers the job to the memory computing system cluster for operation and processing.
When the memory computing system is a Spark computing system, the Sqoop task analyzer generates Sqoop task jobs in a command writing mode supported by a user by using a Sqoop component, analyzes the Sqoop task jobs, analyzes the input and output of the Sqoop jobs and the corresponding relation between a map method and the corresponding input and output supported by Spark and TANSFORM, and then sends the Sqoop task jobs to the Sqoop-Spark converter. Specifically, the user writes a Sqoop command script in advance, that is, writes a Sqoop command according to a Sqoop command rule, and stores the Sqoop command script in a shell file. After a user executes the shell script file and the Sqoop executable task is generated, the Sqoop operation analyzer reads the generated operation, and analyzes the DataInput input, the DataInput output and the map method into the corresponding input method, output method and TANSFORM method in the SparkContext. For example, a method for reading a text file of the HDFS exists in the Sqoop job, and the Sqoop parser parses the text file into textFile methods of SparkContext according to the corresponding relationships, and sends the corresponding relationships to the Sqoop-Spark converter.
The Sqoop-Spark converter obtains the corresponding relation between the Sqoop operation analyzed by the Sqoop operation analyzer and Spark operation, then converts the operation generated by Sqoop into the operation which can be identified by Spark, and compiles and generates jar packets. For example: acquiring the corresponding relationship between the Sqoop job DataInput input, DataInput output and map method and the method in Spark job, for example: in the method for reading the HDFS text file of the Sqoop job, a method function in the sparklContext corresponding to the method function is textFile. And generating a Spark job according to the corresponding relation, for example: according to the above correspondence, the data source type read in Sqoop is HDFS, and a Spark method named textFile is used in Spark job. And after the whole conversion of the operation is finished, compiling and packaging to generate a Spark executable jar package. And the Spark job submitter submits the Spark job jar packet generated by the Sqoop-Spark converter to a Spark cluster to be run.
In the following, the database data transmission device provided by the embodiment of the present invention is introduced, and the database data transmission device described below and the database data transmission method described above may be referred to correspondingly.
Referring to fig. 2, fig. 2 is a schematic structural diagram of a database data transmission device according to an embodiment of the present invention, including:
an obtaining module 110, configured to obtain a first task file for executing a database data transmission task;
a conversion module 120, configured to convert the first task file into a second task file matched with the memory computing system;
the sending module 130 is configured to send the second task file to the memory computing system, so that the memory computing system executes a database data transmission task by using the second task file.
Optionally, the conversion module 120 includes:
an acquisition unit configured to acquire first input information, first output information, and first function information from a first task file;
the conversion unit is used for converting the first input information, the first output information and the first function information according to the format conversion relation to obtain second input information, second output information and second function information which are matched with the memory computing system;
and the second task file generating unit is used for generating a second task file by using the second input information, the second output information and the second function information.
Optionally, the method further comprises:
the information acquisition module is used for acquiring first information and second information matched with the memory computing system;
and the conversion relation establishing module is used for establishing a corresponding relation between the first information and the second information to obtain a format conversion relation.
Optionally, the obtaining module 110 includes:
the information acquisition unit is used for acquiring task information and a command script file;
and the first task file generating unit is used for generating a first task file according to the task information by using the command script file.
Optionally, the method further comprises:
the command generating module is used for generating a plurality of commands according to command rules;
and the script file generating module is used for generating a command script file by using the command.
Optionally, the obtaining module 110 includes:
and the target acquisition unit is used for acquiring the first task file sent by the target equipment.
Optionally, the method further comprises:
and the feedback module is used for feeding back the task information to the target equipment.
In the following, the database data transmission device provided by the embodiment of the present invention is introduced, and the database data transmission device described below and the database data transmission method described above may be referred to in correspondence with each other.
Referring to fig. 3, fig. 3 is a schematic structural diagram of a database data transmission device according to an embodiment of the present invention. Wherein the database data transmission device 100 may include a processor 101 and a memory 102, and may further include one or more of a multimedia component 103, an information input/information output (I/O) interface 104, and a communication component 105.
The processor 101 is configured to control the overall operation of the database data transmission apparatus 100, so as to complete all or part of the steps in the database data transmission method; the memory 102 is used to store various types of data to support operation at the database data transmission device 100, which may include, for example, instructions for any application or method operating on the database data transmission device 100, as well as application-related data. The Memory 102 may be implemented by any type or combination of volatile and non-volatile Memory devices, such as one or more of Static Random Access Memory (SRAM), Electrically erasable Programmable Read-Only Memory (EEPROM), erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic Memory, flash Memory, magnetic disk, or optical disk.
The multimedia component 103 may include a screen and an audio component. Wherein the screen may be, for example, a touch screen and the audio component is used for outputting and/or inputting audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may further be stored in the memory 102 or transmitted through the communication component 105. The audio assembly also includes at least one speaker for outputting audio signals. The I/O interface 104 provides an interface between the processor 101 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 105 is used for wired or wireless communication between the database data transmission device 100 and other devices. Wireless Communication, such as Wi-Fi, bluetooth, Near Field Communication (NFC), 2G, 3G, or 4G, or a combination of one or more of them, so that the corresponding Communication component 105 may include: Wi-Fi part, Bluetooth part, NFC part.
The database data transmission Device 100 may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components, and is used to execute the database data transmission method according to the above embodiments.
In the following, the computer-readable storage medium provided by the embodiment of the present invention is introduced, and the computer-readable storage medium described below and the database data transmission method described above may be referred to correspondingly.
The present invention also provides a computer-readable storage medium, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-mentioned database data transmission method.
The computer-readable storage medium may include: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
The embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same or similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.
Those of skill would further appreciate that the various illustrative components and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
Finally, it should also be noted that, herein, relationships such as first and second, etc., are intended only to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms include, or any other variation is intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.
The database data transmission method, the database data transmission device and the computer readable storage medium provided by the present invention are described in detail above, and specific examples are applied herein to explain the principle and the implementation of the present invention, and the description of the above embodiments is only used to help understand the method and the core idea of the present invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.
Claims (10)
1. A method for database data transmission, comprising:
acquiring a first task file for executing a database data transmission task;
converting the first task file into a second task file matched with a memory computing system;
and sending the second task file to the memory computing system so that the memory computing system executes the database data transmission task by using the second task file.
2. The method for transferring database data according to claim 1, wherein the converting the first task file into a second task file matching a memory computing system comprises:
acquiring first input information, first output information and first function information from the first task file;
converting the first input information, the first output information and the first function information according to a format conversion relation to obtain second input information, second output information and second function information which are matched with the memory computing system;
and generating the second task file by using the second input information, the second output information and the second function information.
3. The database data transmission method according to claim 2, further comprising:
acquiring first information and second information matched with the memory computing system;
and establishing a corresponding relation between the first information and the second information to obtain the format conversion relation.
4. The database data transmission method according to claim 1, wherein the obtaining a first task file for executing a database data transmission task includes:
acquiring task information and a command script file;
and generating the first task file according to the task information by using the command script file.
5. The database data transmission method according to claim 4, further comprising:
generating a plurality of commands according to the command rule;
and generating the command script file by using the command.
6. The database data transmission method according to claim 1, wherein the obtaining a first task file for executing a database data transmission task includes:
and acquiring the first task file sent by the target equipment.
7. The database data transmission method according to claim 6, further comprising:
and feeding back task information to the target equipment.
8. A database data transmission apparatus, comprising:
the system comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring a first task file for executing a database data transmission task;
the conversion module is used for converting the first task file into a second task file matched with the memory computing system;
and the sending module is used for sending the second task file to the memory computing system so that the memory computing system can execute the database data transmission task by using the second task file.
9. A database data transmission apparatus comprising a memory and a processor, wherein:
the memory is used for storing a computer program;
the processor is configured to execute the computer program to implement the database data transmission method according to any one of claims 1 to 7.
10. A computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the database data transmission method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010710074.XA CN111814435A (en) | 2020-07-22 | 2020-07-22 | Database data transmission method, device, equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010710074.XA CN111814435A (en) | 2020-07-22 | 2020-07-22 | Database data transmission method, device, equipment and readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111814435A true CN111814435A (en) | 2020-10-23 |
Family
ID=72861936
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010710074.XA Pending CN111814435A (en) | 2020-07-22 | 2020-07-22 | Database data transmission method, device, equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111814435A (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170075964A1 (en) * | 2015-09-11 | 2017-03-16 | International Business Machines Corporation | Transforming and loading data utilizing in-memory processing |
| US20170097957A1 (en) * | 2015-10-01 | 2017-04-06 | International Business Machines Corporation | System and method for transferring data between rdbms and big data platform |
| CN106569789A (en) * | 2015-10-13 | 2017-04-19 | 北京国双科技有限公司 | Task submission method and apparatus |
| CN106648674A (en) * | 2016-12-28 | 2017-05-10 | 北京奇艺世纪科技有限公司 | Big data computing management method and system |
| CN106777164A (en) * | 2016-12-20 | 2017-05-31 | 东软集团股份有限公司 | A kind of Data Migration cluster and data migration method |
| CN107145585A (en) * | 2017-05-10 | 2017-09-08 | 温州市鹿城区中津先进科技研究院 | The automated import of data method and system of Hadoop data warehouses |
| US20190197142A1 (en) * | 2017-12-21 | 2019-06-27 | Mastercard International Incorporated | Systems and Methods for Use in Loading Data From Backup Databases to Task Databases |
| CN110781235A (en) * | 2019-10-24 | 2020-02-11 | 珠海格力电器股份有限公司 | Big data based purchase data processing method and device, terminal and storage medium |
| CN111198863A (en) * | 2019-12-27 | 2020-05-26 | 天阳宏业科技股份有限公司 | Rule engine and implementation method thereof |
-
2020
- 2020-07-22 CN CN202010710074.XA patent/CN111814435A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170075964A1 (en) * | 2015-09-11 | 2017-03-16 | International Business Machines Corporation | Transforming and loading data utilizing in-memory processing |
| US20170097957A1 (en) * | 2015-10-01 | 2017-04-06 | International Business Machines Corporation | System and method for transferring data between rdbms and big data platform |
| CN106569789A (en) * | 2015-10-13 | 2017-04-19 | 北京国双科技有限公司 | Task submission method and apparatus |
| CN106777164A (en) * | 2016-12-20 | 2017-05-31 | 东软集团股份有限公司 | A kind of Data Migration cluster and data migration method |
| CN106648674A (en) * | 2016-12-28 | 2017-05-10 | 北京奇艺世纪科技有限公司 | Big data computing management method and system |
| CN107145585A (en) * | 2017-05-10 | 2017-09-08 | 温州市鹿城区中津先进科技研究院 | The automated import of data method and system of Hadoop data warehouses |
| US20190197142A1 (en) * | 2017-12-21 | 2019-06-27 | Mastercard International Incorporated | Systems and Methods for Use in Loading Data From Backup Databases to Task Databases |
| CN110781235A (en) * | 2019-10-24 | 2020-02-11 | 珠海格力电器股份有限公司 | Big data based purchase data processing method and device, terminal and storage medium |
| CN111198863A (en) * | 2019-12-27 | 2020-05-26 | 天阳宏业科技股份有限公司 | Rule engine and implementation method thereof |
Non-Patent Citations (2)
| Title |
|---|
| 于金良 等: "一种基于Sqoop的数据交换系统", 《物联网技术》 * |
| 王永超 等: "面向金融行业的大数据迁移的研究与实现", 《计算机工程与应用》 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10423445B2 (en) | Composing and executing workflows made up of functional pluggable building blocks | |
| US9244709B2 (en) | Automatic recognition of web application | |
| US20140372970A1 (en) | Method to auto generate jax-rs rest service implementation classes from existing interfaces | |
| US9892144B2 (en) | Methods for in-place access of serialized data | |
| US20110295865A1 (en) | Schema Contracts for Data Integration | |
| US20140359586A1 (en) | Programming Language with Extensions using a Strict Meta-Model | |
| CN108255837B (en) | SQL parser and method | |
| CN109947431B (en) | Code generation method, device, equipment and storage medium | |
| WO2016095570A1 (en) | Debugging method and apparatus for embedded system, and storage medium | |
| US9141344B2 (en) | Hover help support for application source code | |
| CN114036183A (en) | Data ETL processing method, device, equipment and medium | |
| CN112988873A (en) | Data processing method and device | |
| CN116028028A (en) | Request function generation method, device, equipment and storage medium | |
| CN113806429A (en) | Canvas type log analysis method based on large data stream processing framework | |
| CN105867886B (en) | Method and device for writing table | |
| CN108153896B (en) | Processing method and device for input data and output data | |
| US20150339106A1 (en) | Tabular format transaction editor | |
| CN111814435A (en) | Database data transmission method, device, equipment and readable storage medium | |
| CN116893809A (en) | Method, storage medium and system for code enrichment of code interpretability | |
| KR20200103133A (en) | Method and apparatus for performing extract-transfrom-load procedures in a hadoop-based big data processing system | |
| WO2021259290A1 (en) | Stored procedure conversion method and apparatus, and device and storage medium | |
| US9015679B2 (en) | System and method for translating business application functions into DBMS internal programming language procedures | |
| US8108828B2 (en) | System for generating optimized computer data field conversion routines | |
| CN111151008B (en) | Verification method and device for game operation data, configuration background and medium | |
| US9720660B2 (en) | Binary interface instrumentation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20201023 |
|
| RJ01 | Rejection of invention patent application after publication |