CN111177267A - Data transmission method and device, electronic equipment and storage medium - Google Patents
Data transmission method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN111177267A CN111177267A CN201911399587.7A CN201911399587A CN111177267A CN 111177267 A CN111177267 A CN 111177267A CN 201911399587 A CN201911399587 A CN 201911399587A CN 111177267 A CN111177267 A CN 111177267A
- Authority
- CN
- China
- Prior art keywords
- data transmission
- data packet
- data
- fingerprint
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/602—Providing cryptographic facilities or services
Abstract
The invention discloses a data transmission method, which comprises the following steps: acquiring a data packet from a source database, and taking an encrypted server timestamp as a fingerprint of the data packet; comparing the fingerprint with a fingerprint prestored in a target database, and if the fingerprint is consistent with the fingerprint prestored in the target database, writing the content of the data packet into the target database; and if the data transmission rate is inconsistent, terminating the data transmission. The method can avoid the phenomenon of packet loss in the transmission process of the data and improve the reliability of data transmission between heterogeneous databases.
Description
Technical Field
The invention relates to the technical field of internet, in particular to a data transmission method and device.
Background
With the continuous development of internet services, the demands of various business departments of a company on business data are gradually increased, such as data analysis, report display and the like, and the databases used by the various departments are different in order to adapt to the business demands of the departments.
In order to realize data communication among various heterogeneous databases, the most original method is to extract service data from each database to generate an excel table, and then send the excel table to each department, or use an offline data synchronization tool/platform to transmit data in the heterogeneous databases to each other. A large amount of service data generates a large amount of inter-database operations, including multiple steps of backup, transmission, recovery and the like, which are prone to misoperation and data loss and the like.
For the reliability problem of data transmission of the off-line data synchronization platform, no countermeasure is provided at the present stage.
Disclosure of Invention
Based on the above, the invention provides a data transmission method and device, which can ensure that data is not lost in the data transmission process, and are convenient for improving the reliability of data transmission.
In a first aspect of the present invention, a data transmission method is provided, where the method includes:
acquiring a data packet from a source database, and taking an encrypted server timestamp as a fingerprint of the data packet;
comparing the fingerprint with a fingerprint prestored in a target database, and if the fingerprint is consistent with the fingerprint prestored in the target database, writing the content of the data packet into the target database; and if the data transmission rate is inconsistent, terminating the data transmission.
Specifically, the method further comprises:
pre-checking the data packet when acquiring the data packet from the source database, judging whether the data packet contains the existing field of the target database,
if yes, comparing the value corresponding to the field in the data packet with the value corresponding to the field of the target database, if the values are the same, terminating the data transmission, if the values are different, continuing the data transmission,
and if not, continuing the data transmission.
Specifically, the method further comprises:
and decomposing the data packet into sub-packets, and grouping the sub-packets according to the number of channels and/or threads which are arbitrarily set for transmission.
Specifically, the method further comprises:
the source database and the destination database independently comprise: mysql, Oracle, DRDS, Hive, or SQLServer.
Specifically, the method further comprises:
the fingerprint is 128 bits and is obtained by MD5 encryption.
A second aspect of the present invention provides a data transmission apparatus, including:
the database writing module is used for acquiring a data packet from a source database and taking the encrypted server timestamp as the fingerprint of the data packet;
the check-before-write module is used for comparing the fingerprint with a fingerprint prestored in a target database, and if the fingerprint is consistent with the fingerprint prestored in the target database, writing the content of the data packet into the target database; and if the data transmission rate is inconsistent, terminating the data transmission.
Specifically, the apparatus further comprises:
the pre-checking module is used for pre-checking the data packet when the data packet is acquired from the source database, judging whether the data packet contains the existing field of the target database, if so, comparing the value corresponding to the field in the data packet with the value corresponding to the field of the target database, and if so, terminating the data transmission; if not, continuing the data transmission,
and if not, continuing the data transmission.
Specifically, the apparatus further comprises:
and the distribution module is used for decomposing the data packet into sub-data packets and grouping and transmitting the sub-data packets according to the number of channels and/or threads which are set randomly.
In a third aspect of the invention, a terminal is provided, which includes a memory for storing a computer program, a processor; the processor is arranged to run the computer program to perform the steps of the data transmission method of the invention.
A fourth aspect of the present invention provides a computer-readable storage medium on which a computer program is stored, characterized in that the program, when executed by a processor, implements the steps of the data transmission method of the present invention.
In conclusion, the invention can avoid packet loss in the data transmission process and effectively improve the reliability of data transmission between heterogeneous databases or homogeneous databases by generating the fingerprint in the data transmission stage and verifying whether the transmitted data is continuous or not by using the fingerprint.
Drawings
In order to more clearly illustrate the embodiments of the present specification or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only some embodiments described in the present specification, and for those skilled in the art, other drawings can be obtained according to the drawings without any creative effort.
Fig. 1 is a schematic flow chart illustrating a data transmission method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a data transmission apparatus according to another embodiment of the present invention;
fig. 3 is a flowchart of a data transmission terminal according to another embodiment of the present invention;
fig. 4 is a schematic structural diagram of a computer-readable storage medium according to another embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the drawings in the embodiments of the present disclosure, and it is obvious that the described embodiments are only a part of the embodiments of the present disclosure, and not all of the embodiments. All other embodiments obtained by a person skilled in the art based on the embodiments in the present specification without any inventive step should fall within the scope of protection of the present specification.
Example 1
Fig. 1 is a schematic flow chart of a data transmission method provided in the present application. The method comprises the following steps:
acquiring a data packet from a source database, and taking an encrypted server timestamp as a fingerprint of the data packet;
comparing the fingerprint with a fingerprint prestored in a target database, and if the fingerprint is consistent with the fingerprint prestored in the target database, writing the content of the data packet into the target database; and if the data transmission rate is inconsistent, terminating the data transmission.
The fingerprint prestored in the target database refers to the fingerprint of a data packet, each data packet fingerprint is generated by encrypting a server timestamp, a specific encryption algorithm is a summary encryption algorithm, and the fingerprint of the first data packet is an initialization fingerprint generated when a data transmission channel is initialized.
When the fingerprints are inconsistent, the data packet loss problem is solved, and then data transmission is stopped. The step can effectively prevent the data packet loss from occurring. The source database and the target database in this step may be the same type of database or different types of databases.
The data packet is a unit for data transfer into which a data transfer job is divided when the data transfer job occurs.
Preferably, the method further comprises:
pre-checking the data packet when acquiring the data packet from the source database, judging whether the data packet contains the same field as the target database,
if yes, comparing the value corresponding to the field in the data packet with the value corresponding to the field of the target database, and if the values are the same, terminating the data transmission; if not, continuing the data transmission,
and if not, continuing the data transmission.
Preferably, the method further comprises:
and decomposing the data packet into sub-packets, and grouping the sub-packets according to the number of channels and/or threads which are arbitrarily set for transmission.
The data synchronization tool receives the data synchronization operation and starts a process to complete the whole operation synchronization process. The tool is a central control node of a single operation and has the functions of data cleaning, subtask segmentation (converting single operation calculation into a plurality of sub tasks), Task group management and the like.
For example, a user submitted a job and configured 20 concurrencies to synchronize mysql data for a 100-sheet spreadsheet to odps. The tool scheduling decision idea is as follows:
1. the table is divided into 100 tasks according to the database and table.
2. The DataX calculation needs to allocate 4 taskgroups in total, according to 20 concurrencies.
And 3.4 Taskgroups equally divide the split 100 Tasks, and each Taskgroup is responsible for running 25 Tasks in a total of 5 concurrences.
Preferably, the method further comprises:
the source database and the destination database independently comprise: mysql, Oracle, DRDS, Hive, or SQLServer.
Preferably, the method further comprises:
the fingerprint is 128 bits and is obtained by MD5 encryption.
The MD5 encryption can calculate an input string with any length to obtain an output with a fixed length, so that the fingerprint format is unified conveniently, and meanwhile, the encryption method has rapid development and high reliability.
Example 2
A data transmission apparatus, characterized in that the apparatus comprises:
the database writing module is used for acquiring a data packet from a source database and taking the encrypted server timestamp as the fingerprint of the data packet;
the check-before-write module is used for comparing the fingerprint with a fingerprint prestored in a target database, and if the fingerprint is consistent with the fingerprint prestored in the target database, writing the content of the data packet into the target database; and if the data transmission rate is inconsistent, terminating the data transmission.
When the fingerprints are inconsistent, the data packet loss problem is solved, and then data transmission is stopped. The step can effectively prevent the data packet loss from occurring. The source database and the target database in the comparison step can be the same type of database or different types of databases.
The data packet is a unit for data transfer into which a data transfer job is divided when the data transfer job occurs.
Preferably, the apparatus further comprises:
the pre-checking module is used for pre-checking the data packet when the data packet is acquired from the source database, judging whether the data packet contains the existing field of the target database, if so, comparing the value corresponding to the field in the data packet with the value corresponding to the field of the target database, and if so, terminating the data transmission; if not, continuing the data transmission,
and if not, continuing the data transmission.
Preferably, the apparatus further comprises:
and the distribution module is used for decomposing the data packet into sub-data packets and grouping and transmitting the sub-data packets according to the number of channels and/or threads which are set randomly.
The data synchronization tool receives the data synchronization operation and starts a process to complete the whole operation synchronization process. The tool is a central control node of a single operation and has the functions of data cleaning, subtask segmentation (converting single operation calculation into a plurality of sub tasks), Task group management and the like.
For example, a user submitted a job and configured 20 concurrencies to synchronize mysql data for a 100-sheet spreadsheet to odps. The tool scheduling decision idea is as follows:
1. the table is divided into 100 tasks according to the database and table.
2. The DataX calculation needs to allocate 4 taskgroups in total, according to 20 concurrencies.
And 3.4 Taskgroups equally divide the split 100 Tasks, and each Taskgroup is responsible for running 25 Tasks in a total of 5 concurrences.
Preferably, the present device generates the packet fingerprint using MD5 encryption.
The MD5 encryption can calculate an input string with any length to obtain an output with a fixed length, so that the fingerprint format is unified conveniently, and meanwhile, the encryption method has rapid development and high reliability.
Example 3
A terminal comprising a memory, a processor, the memory for storing a computer program; the processor is arranged to run the computer program to perform the steps of the data transmission method of the invention.
As shown in fig. 3, the terminal 2 includes: a processor 202, a memory 201, a bus 204, and a computer program 203, wherein:
the terminal may comprise one or more processors 202, a bus 204 and a memory 201, wherein the memory 201 is used for storing programs, the processor 202 is communicatively connected with the memory 201 through the bus 204, and the processor 202 calls the computer program 203 stored by the memory 201 to execute the above-mentioned method embodiments.
It is noted that processor 202 may include one or more processing cores (e.g., a single-core processor or a multi-core processor). Merely by way of example, a Processor may include a Central Processing Unit (CPU), an Application Specific Integrated Circuit (ASIC), an Application Specific Instruction Set Processor (ASIP), a Graphics Processing Unit (GPU), a Physical Processing Unit (PPU), a Digital Signal Processor (DSP), a Field Programmable Gate Array (FPGA), a Programmable Logic Device (PLD), a controller, a microcontroller Unit, a Reduced Instruction Set computer (Reduced Instruction Set computer), a microprocessor, or the like, or any combination thereof.
The memory 201 may include: including mass storage, removable storage, volatile Read-and-write Memory, or Read-Only Memory (ROM), among others, or any combination thereof. By way of example, mass storage may include magnetic disks, optical disks, solid state drives, and the like; removable memory may include flash drives, floppy disks, optical disks, memory cards, zip disks, tapes, and the like; volatile read-write Memory may include Random Access Memory (RAM); the RAM may include Dynamic RAM (DRAM), double data Rate Synchronous Dynamic RAM (DDR SDRAM); static RAM (SRAM), Thyristor-Based Random Access Memory (T-RAM), Zero-capacitor RAM (Zero-RAM), and the like. By way of example, ROMs may include Mask Read-Only memories (MROMs), Programmable ROMs (PROMs), Erasable Programmable ROMs (PERROMs), Electrically Erasable Programmable ROMs (EEPROMs), compact disk ROMs (CD-ROMs), digital versatile disks (ROMs), and the like.
Example 4
A computer-readable storage medium, as shown in fig. 4, on which a computer program is stored, characterized in that the program, when executed by a processor, implements the steps of the data transmission method of the present invention.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
The data transmission method and the data transmission device provided by the present application are described in detail above. The principles and embodiments of the present application are explained herein using specific examples, which are provided only to help understand the method and the core idea of the present application. It should be noted that, for those skilled in the art, it is possible to make several improvements and modifications to the present application without departing from the principle of the present application, and such improvements and modifications also fall within the scope of the claims of the present application.
Claims (10)
1. A method of data transmission, comprising:
acquiring a data packet from a source database, and taking an encrypted server timestamp as a fingerprint of the data packet;
comparing the fingerprint with a fingerprint prestored in a target database, and if the fingerprint is consistent with the fingerprint prestored in the target database, writing the content of the data packet into the target database; and if the data transmission rate is inconsistent, terminating the data transmission.
2. The data transmission method of claim 1, further comprising:
pre-checking the data packet when acquiring the data packet from the source database, judging whether the data packet contains the existing field of the target database,
if yes, comparing the value corresponding to the field in the data packet with the value corresponding to the existing field of the target database, and if the values are the same, terminating the data transmission; if not, continuing the data transmission,
and if not, continuing the data transmission.
3. The data transmission method of claim 2, further comprising:
and decomposing the data packet into sub-packets, and grouping the sub-packets according to the number of channels and/or threads which are arbitrarily set for transmission.
4. The data transmission method of claim 3, further comprising:
the source database and the destination database independently comprise: mysql, Oracle, DRDS, Hive, or SQLServer.
5. The data transmission method of claim 4, further comprising:
the fingerprint is 128 bits and is obtained by MD5 encryption.
6. A data transmission apparatus, characterized in that the apparatus comprises:
the database writing module is used for acquiring a data packet from a source database and taking the encrypted server timestamp as the fingerprint of the data packet;
the check-before-write module is used for comparing the fingerprint with a fingerprint prestored in a target database, and if the fingerprint is consistent with the fingerprint prestored in the target database, writing the content of the data packet into the target database; and if the data transmission rate is inconsistent, terminating the data transmission.
7. The data transmission apparatus of claim 6, wherein the apparatus further comprises:
the pre-checking module is used for pre-checking the data packet when the data packet is acquired from the source database, judging whether the data packet contains the existing field of the target database, if so, comparing the value corresponding to the field in the data packet with the value corresponding to the field of the target database, and if so, terminating the data transmission; if not, continuing the data transmission,
and if not, continuing the data transmission.
8. The data transmission apparatus of claim 7, wherein the apparatus further comprises:
and the distribution module is used for decomposing the data packet into sub-data packets and grouping and transmitting the sub-data packets according to the number of channels and/or threads which are set randomly.
9. A terminal comprising a memory, a processor, the memory for storing a computer program; the processor is arranged to run the computer program to perform the steps of the data transmission method of any of claims 1 to 5.
10. A computer-readable storage medium, on which a computer program is stored, which program, when being executed by a processor, carries out the steps of the data transmission method according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911399587.7A CN111177267A (en) | 2019-12-30 | 2019-12-30 | Data transmission method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911399587.7A CN111177267A (en) | 2019-12-30 | 2019-12-30 | Data transmission method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111177267A true CN111177267A (en) | 2020-05-19 |
Family
ID=70654266
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911399587.7A Pending CN111177267A (en) | 2019-12-30 | 2019-12-30 | Data transmission method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111177267A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112019575A (en) * | 2020-10-22 | 2020-12-01 | 腾讯科技（深圳）有限公司 | Data packet processing method and device, computer equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140101444A1 (en) * | 2012-10-09 | 2014-04-10 | Samsung Sds Co., Ltd. | Apparatus and method for transmitting data, and recording medium storing program for executing method of the same in computer |
| CN106850377A (en) * | 2017-03-29 | 2017-06-13 | 北京润科通用技术有限公司 | The check system and method for a kind of AFDX bus messages |
| CN108281174A (en) * | 2018-02-24 | 2018-07-13 | 量化医学研究院（深圳）有限公司 | A kind of data interconnection method and data docking system |
| CN108833080A (en) * | 2018-06-05 | 2018-11-16 | 中国联合网络通信集团有限公司 | A kind of data transmission method, device and network system |
-
2019
- 2019-12-30 CN CN201911399587.7A patent/CN111177267A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140101444A1 (en) * | 2012-10-09 | 2014-04-10 | Samsung Sds Co., Ltd. | Apparatus and method for transmitting data, and recording medium storing program for executing method of the same in computer |
| CN106850377A (en) * | 2017-03-29 | 2017-06-13 | 北京润科通用技术有限公司 | The check system and method for a kind of AFDX bus messages |
| CN108281174A (en) * | 2018-02-24 | 2018-07-13 | 量化医学研究院（深圳）有限公司 | A kind of data interconnection method and data docking system |
| CN108833080A (en) * | 2018-06-05 | 2018-11-16 | 中国联合网络通信集团有限公司 | A kind of data transmission method, device and network system |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112019575A (en) * | 2020-10-22 | 2020-12-01 | 腾讯科技（深圳）有限公司 | Data packet processing method and device, computer equipment and storage medium |
| CN112019575B (en) * | 2020-10-22 | 2021-01-29 | 腾讯科技（深圳）有限公司 | Data packet processing method and device, computer equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| RU2724136C1 (en) | Data processing method and device | |
| US10209908B2 (en) | Optimization of in-memory data grid placement | |
| CN110334075B (en) | Data migration method based on message middleware and related equipment | |
| US8898422B2 (en) | Workload-aware distributed data processing apparatus and method for processing large data based on hardware acceleration | |
| Xie et al. | Pandas: robust locality-aware scheduling with stochastic delay optimality | |
| US9535743B2 (en) | Data processing control method, computer-readable recording medium, and data processing control device for performing a Mapreduce process | |
| WO2017107812A1 (en) | User log storage method and device | |
| CN107085570A (en) | Data processing method, application server and router | |
| CN104518987A (en) | Method and device for processing parallel multithreading messages | |
| CN107402956A (en) | Data processing method, equipment and the computer-readable recording medium of big task | |
| CN111522811B (en) | Database processing method and device, storage medium and terminal | |
| CN112416654A (en) | Database log replay method, device, equipment and storage medium | |
| CN107153680B (en) | Method and system for on-line node expansion of distributed memory database | |
| CN104281636B (en) | The concurrent distributed approach of magnanimity report data | |
| CN106909556A (en) | The storage equalization methods and device of main memory cluster | |
| CN111177267A (en) | Data transmission method and device, electronic equipment and storage medium | |
| WO2021174882A1 (en) | Data fragment verification method, apparatus, computer device, and readable storage medium | |
| US9578120B1 (en) | Messaging with key-value persistence | |
| Julián-Moreno et al. | On the feasibility of 40 gbps network data capture and retention with general purpose hardware | |
| JP6899972B2 (en) | Information processing equipment, information processing methods and information processing programs | |
| CN115951845A (en) | Disk management method, device, equipment and storage medium | |
| CN108234552B (en) | Data storage method and device | |
| KR102416336B1 (en) | Device, method, system and computer readable storage medium for managing blockchain | |
| US10540341B1 (en) | System and method for dedupe aware storage quality of service | |
| CN109522915B (en) | Virus file clustering method and device and readable medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |