CN108347477B - Data transmission method, device and server - Google Patents
Data transmission method, device and server Download PDFInfo
- Publication number
- CN108347477B CN108347477B CN201810069141.7A CN201810069141A CN108347477B CN 108347477 B CN108347477 B CN 108347477B CN 201810069141 A CN201810069141 A CN 201810069141A CN 108347477 B CN108347477 B CN 108347477B
- Authority
- CN
- China
- Prior art keywords
- data
- sent
- server
- ordered
- message queue
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/60—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The embodiment of the application discloses a data sending method, a data sending device and a server. The method comprises the following steps: the server receives a plurality of data to be sent, wherein each data to be sent comprises an ordered mark; dividing the data to be sent into one or more data packets according to the order marks in the data to be sent, wherein the data to be sent with the same order mark is divided into the same data packet, and the data to be sent with different order marks is divided into different data packets; and placing the one or more data packets into at least one of the plurality of message queues, wherein the data to be sent in the same data packet is placed into the same message queue according to the receiving sequence. The method ensures that when the server puts a plurality of data packets into a plurality of message queues, the transmission in the data synchronization process can be carried out in a multilink manner, thereby improving the synchronization efficiency.
Description
Technical Field
The present application relates to the field of data synchronization, and more particularly, to a data sending method, device and server.
Background
Some internet sites deploy multiple sites at different locations for the purpose of data security backup or improving access speed of users at various locations. Wherein each site is a server or a cluster of multiple servers. Data synchronization is maintained between the sites so that part of the data content or all of the data content stored therein is consistent.
In data synchronization, in order to ensure the sequentiality of synchronization, data is synchronized in a single-thread mode, and the synchronization efficiency is low.
Disclosure of Invention
In view of the above problems, the present application provides a data transmission method, an apparatus and a server to solve the problem of low synchronization efficiency in the data synchronization process.
In a first aspect, the present application provides a data sending method, which is applied to a cross-machine-room data synchronization system, where the cross-machine-room data synchronization system includes a message queue device and a server; the message queue device comprises a plurality of message queues, the server is used for processing data to be sent and then putting the processed data into at least one of the message queues, and the message queues are first-in first-out queues used for sending the data; the method comprises the following steps: the server receives a plurality of data to be sent, wherein each data to be sent comprises an ordered mark; dividing the data to be sent into one or more data packets according to the order marks in the data to be sent, wherein the data to be sent with the same order mark is divided into the same data packet, and the data to be sent with different order marks is divided into different data packets; and placing the one or more data packets into at least one of the plurality of message queues, wherein the data to be sent in the same data packet is placed into the same message queue according to the receiving sequence.
In a second aspect, the present application provides a data sending device, which operates in a server in a cross-machine-room data synchronization system, where the cross-machine-room data synchronization system includes a message queue device and a server; the message queue device comprises a plurality of message queues, the server is used for processing data to be sent and then putting the processed data into at least one of the message queues, and the message queues are first-in first-out queues used for sending the data; the data transmission apparatus includes: the data receiving module is used for receiving a plurality of data to be sent, wherein each data to be sent comprises an ordered mark; the data dividing module is used for dividing the data to be sent into one or more data packets according to the ordered marks in the data to be sent, wherein the data to be sent with the same ordered mark is divided into the same data packet, and the data to be sent with different ordered marks is divided into different data packets; and the enqueuing module is used for putting the one or more data packets into at least one message queue of the plurality of message queues, wherein the data to be sent in the same data packet are put into the same message queue according to the receiving sequence.
In a third aspect, the present application provides a server comprising one or more processors and a memory; one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs configured to perform the methods described above.
In a fourth aspect, the present application provides a computer readable storage medium having program code executable by a processor, the program code causing the processor to perform the method described above.
According to the data sending method, the data sending device and the server, in the cross-machine-room data synchronization system, the message queue device comprises a plurality of first-in first-out message queues for sending data, and after the server receives the data to be sent, the server divides the data packets according to the ordered marks in the data to be sent, and divides the data to be sent into one or more data packets. And the one or more data packets are put into at least one message queue of the plurality of message queues, so when the plurality of data packets are put into the plurality of message queues, the data are sent in parallel from the plurality of message queues, the sending efficiency is improved, and the data synchronization efficiency in the cross-machine room data synchronization system is improved.
These and other aspects of the present application will be more readily apparent from the following description of the embodiments.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 is a schematic diagram illustrating an environment of a cross-room data synchronization system proposed in the present application;
fig. 2 shows a block diagram of a cross-room data synchronization system proposed in the present application;
fig. 3 is a flow chart of a data transmission method proposed in the present application;
FIG. 4 is a flow chart of another method of sending data as proposed by the present application;
fig. 5 is a block diagram showing a structure of a data transmission apparatus proposed in the present application;
fig. 6 shows a block diagram of a server for executing a data transmission method according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Usually, an internet company deploys a plurality of sites distributed in different areas, and sometimes data synchronization is performed between different sites in order to ensure that data between the deployed different sites is the same or partial data in the sites is the same. For example, in order to facilitate the user to log in when accessing each site, login information such as user names and passwords of all users may be synchronized between different sites.
Usually, a site is built in a computer room, so when data synchronization is performed between different sites, it can also be regarded as data synchronization between different computer rooms. The inventor finds that in data synchronization, data is generally synchronized in a single-thread mode in order to guarantee the sequentiality of synchronization, and in the single-thread synchronization process, data is transmitted through a single link, so that the synchronization rate is low.
Therefore, the inventor proposes a data sending method in the present application, in which received data to be sent is divided according to an ordered flag, and if the ordered flag is multiple, the data to be sent can be divided into multiple data packets, and the multiple data packets can be placed into multiple message queues, so that the data can be sent in order from the multiple message queues, and thus, data sending can be performed by multiple threads and multiple links, and sending efficiency is improved.
The following first introduces an environment of a cross-machine-room data synchronization system in which the data sending method and apparatus provided by the present application operate.
As shown in fig. 1, the cross-room data synchronization system 100 shown in fig. 1 includes a first room 110 and a second room 120. The first computer room 110 and the second computer room 120 perform data interaction through the network 130. A server 111 and a message queue means 112 in the first room 110 are deployed in the first room 110. A data migration device 121, a message queue apparatus 122, and a server 123 are disposed in the second machine room 120. It should be noted that, as shown in fig. 2, the message queue apparatus 112 and the message queue apparatus 122 may include a plurality of message queues during the data synchronization process, and the message queues include data packets during the synchronization process. The message queues in the message queue apparatus 112 and the message queue apparatus 122 may have the same number, and correspond to each other one by one, and the data packets in the message queue of the message queue apparatus 112 are equivalently migrated into the message queue of the message queue apparatus 122 by the data migration device.
The message queue is a channel for transmitting messages between applications by the applications. Applications communicate by writing and retrieving data (messages) to and from the queue for the application without requiring a dedicated connection to link them. Common message queues are rabbitmq, rocktmq, kafka, and the like.
Referring to fig. 2 again, in the first computer room 110, an SDK (software development kit) component runs in the server 111, and the SDK component may subscribe to service data generated by the data producer and write the service data into the message queue device 112 in the first computer room 110. In the second computer room 120, the data migration device 121 may migrate the data of the message queue apparatus 112 in the first computer room 110 to the message queue apparatus 122, and the server 123 may read the data from the message queue apparatus 122 for processing. The service side sends the data into the SDK by accessing the SDK provided by the data synchronization, and then the service side can finish the data migration to the message queue device of the target machine room. If the service party needs to synchronize the mysql and the redis data, only the addresses of the corresponding mysql and redis data sources need to be provided, and the formed service program is already provided and can be directly used. As one mode, a management node and a plurality of processing nodes run in the server 123, for example, a sender node in fig. 2 is the management node, and a receive node is the processing node. The ETCD is responsible for electing the management node as a processing node and coordinating a plurality of processing nodes to process data in the message queue.
Next, a data transmission flow in the following synchronization process will be described by taking the synchronization of data from the first machine room 110 to the second machine room 120 as an example. In this case, the first machine room 110 is a source machine room of a service data transmitting side, and the second machine room 120 is a destination machine room of a service data receiving side. The SDK component operated by the server 111 in the first computer room 110 encapsulates the received service data into a data packet, and writes the data packet into a message queue in the message queue device 112 in an asynchronous thread manner. The data migration device 121 of the target computer room subscribes to the data in the message queue device 112 in advance, and when the data migration device 121 detects that the message queue in the message queue device 112 has a data packet written therein, migrates the data packet in the message queue into the message queue in the message queue device 122 of the second computer room 120. The server 123 of the second machine room 120 reads the target data packet to be processed from the message queue in the message queue device 122 for processing, thereby completing the synchronization of the data from the first machine room 110 to the second machine room 120. The processing of the data packet includes checking the validity of the data packet and writing the information content carried by the data packet into the local server 123.
It should be noted that the message queue apparatus 112 may be carried by one physical device, for example, one server, or may be carried by a plurality of physical devices, for example, a plurality of servers. The same is true for the message queue means 122. Furthermore, when the data migration apparatus 121 is implemented by a server and the message queue device 122 is also carried by a physical apparatus, the message queue device 122 may be deployed in the data migration apparatus 121, that is, the functions of the data migration apparatus 121 and the message queue device 122 are implemented by a server.
Embodiments of the present application will be described in detail below with reference to the accompanying drawings.
Referring to fig. 3, a data sending method provided in the present application is applied to a cross-room data synchronization system, and the embodiment of the present application is mainly applied to a first room in the system. Specifically, as described above, the cross-room data synchronization system includes a message queue device and a server; the message queue device comprises a plurality of message queues, the server is used for processing data to be sent and then putting the processed data into at least one of the message queues, and the message queues are first-in first-out queues used for sending the data; the method comprises the following steps:
step S110: the server receives a plurality of data to be sent, wherein each data to be sent comprises an ordered mark.
The server receives data to be sent, and each piece of data to be sent includes an ordered label corresponding to the data to be sent, and different pieces of data to be sent may have the same ordered label or different ordered labels.
Step S120: and dividing the data to be sent into one or more data packets according to the order marks in the data to be sent, wherein the data to be sent with the same order mark is divided into the same data packet, and the data to be sent with different order marks is divided into different data packets.
And dividing a plurality of data to be transmitted to obtain data packets which can be transmitted in parallel. Specifically, a plurality of data to be transmitted are divided according to the ordered marks in the data to be transmitted, the data to be transmitted with the same ordered marks are divided into the same data packet, and the data to be transmitted with different ordered marks are divided into different data packets. And if the plurality of ordered labels corresponding to the plurality of data to be sent are the same, obtaining a data packet after segmentation. If there are different ordered labels in the ordered labels corresponding to the multiple data to be sent, the number of the data packets obtained after the division is greater than one, and specifically the number of the obtained data packets is equal to the different number of the ordered labels, for example, if there are A, B, C different ordered labels in the multiple data to be sent, the number of the data packets obtained after the division is 3, where the data in one data packet is the data to be sent with the ordered label a, the data in another data packet is the data to be sent with the ordered label B, and the data in another data packet is the data to be sent with the ordered label C.
Step S130: and placing the one or more data packets into at least one of the plurality of message queues, wherein the data to be sent in the same data packet is placed into the same message queue according to the receiving sequence.
And putting the data packet into a message queue of the message queue device. And if the data packet obtained by dividing is one data packet, the data packet is put into one of a plurality of message queues. If the data to be sent is divided into a plurality of data packets, one or more data packets are respectively put into more than one message queue, that is, the plurality of data packets are respectively put into the message queue, and more than one message queue with the data packets is put into the message queue.
According to the data sending method, the server divides a plurality of data to be sent according to the order marks in the received data to be sent, the data to be sent with the same order marks are divided into the same data packet, the data to be sent with different order marks are divided into different data packets, and therefore the corresponding number of data packets are obtained according to the different numbers of the order marks. And when the number of the data packets is more than one and the number of the message queues for placing the data packets is more than one, the number of the message queues for sending the data in the data packets is more than one, so that the multi-thread parallel sending of the messages is realized, and the sending efficiency is improved.
Referring to fig. 4, a data sending method provided in an embodiment of the present application is applied to a cross-machine-room data synchronization system, where the cross-machine-room data synchronization system includes a message queue device and a server; the message queue device comprises a plurality of message queues, the server is used for processing data to be sent and then putting the processed data into at least one of the message queues, and the message queues are first-in first-out queues used for sending the data; the method comprises the following steps:
step S210: the server receives a plurality of data to be sent, wherein each data to be sent comprises an ordered mark.
The data to be sent received by the server may be service data generated by a producer subscribed by the server, and may reach the server through one link, where the data to be sent of the link reaching the server is ordered, that is, the data to be sent on the link reaches the server in different orders.
In all data arriving at the server, part of the data needs to be sent in order, namely, the data is orderly synchronized to the target computer room, and the synchronization sequence among the part of the data is not limited. Each data to be sent has an ordered mark, and the data to be sent with the same ordered mark are data needing ordered synchronization.
The ordered tag may be a field in a data structure of the data to be sent, e.g., the ordered tag may be a routing field (routingKey). Specifically, the service party may generate service data with a routing field, for example, a payment service is currently, and according to the characteristics of the synchronization data, the order number may be used as the routing field, and the data of the same order number may have the same routing field, so that the ordering of the same order number is realized. If the routing field is not set, the routing field is set according to the table by default, namely, the data in the same table has the same routing field, and different tables have different routing fields.
Step S220: and respectively acquiring the data to be transmitted with the same ordered mark in the plurality of data to be transmitted.
Step S230: and arranging the data to be sent with the same ordered marks according to the receiving sequence to be used as a data packet, and obtaining one or more data packets.
And dividing the received data to be transmitted into one or more data packets according to the ordered marks. First, data with the same ordered labels may be obtained from the received multiple pieces of data to be sent, for example, if the corresponding ordered labels include A, B, C in the multiple pieces of data to be sent, the data to be sent with the ordered labels a, the data to be sent with the ordered labels B, and the data to be sent with the ordered labels C are obtained respectively.
The specific obtaining manner may be to directly obtain the data to be sent corresponding to each ordered tag from the data to be sent of one ordered link that is received. Or, the data to be sent of one ordered link is scattered, and then the data to be sent with the same ordered mark is respectively obtained from the scattered data to be sent.
Secondly, the acquired data to be transmitted with the same ordered mark can be arranged according to the receiving sequence and packaged. The data to be sent corresponding to the same ordered tag is encapsulated into a data packet, and the encapsulated data packet may be added with an ordered tag, where the ordered tag is an ordered tag corresponding to the data packet. For example, the ordered flag is a routing field, the order number in the data is set as the routing field, and when the data is scattered and reassembled, the data containing the same order number in the data is arranged in the front-back order and packaged into a data packet corresponding to the order number.
Step S240: and placing the one or more data packets into at least one of the plurality of message queues, wherein the data to be sent in the same data packet is placed into the same message queue according to the receiving sequence.
And putting the obtained data packet into a message queue in a message queue device. When the data packets are put into the message queue, the data to be sent are put into the same message queue according to the encapsulation sequence of the data packets, that is, the sequence of the data to be sent when the server receives the data to be sent.
If the number of the obtained data packets is more than one, the data packets can be put into more than one message queue, so that the multilink data synchronization is realized through a plurality of message queues. Specifically, if the number of the data packets is less than or equal to the number of the message queues, the data packets may be respectively put into different message queues, and if the number of the data packets is greater than the number of the message queues, the number of the data packets put into each message queue may be balanced as much as possible, so that the message queues send the data packets in a balanced manner, and the synchronization efficiency is improved.
Specifically, the number difference between the data packets put in each of the plurality of message queues does not exceed one, that is, the number difference between the data packets put in any two of the message queues does not exceed one, so that the data is more evenly hashed to each of the message queues.
For example, the packet may be placed in the message queue by the fragmentation algorithm queueName ═ crc16 (routingKey)% 256. The message queues can be numbered according to the number, if 256 message queues exist, the message queues are numbered from 0 to 255, and the queueName represents the number of the message queue. The ordered labels are represented by sequence numbers, such as 1, 2, 3, and so on, and the routingkey represents the ordered label corresponding to the data packet. And obtaining a remainder value after the number of the message queues is measured and the remainder value is obtained according to the ordered marks of the data packets, and taking the message queue corresponding to the number which is equal to the remainder value as the message queue to which the data packets need to be put. If the data packet corresponding to the ordered marker 257 is left for the number 256 of the message queues, the obtained value is 1, and the data packet is put into the message queue with the number of 1. The routing field can realize dynamic change, thereby realizing dynamic switching of the routing in the synchronization process and facilitating the service scene change of the service.
According to the data transmission method provided by the embodiment of the application, the server encapsulates the received data to be transmitted into the corresponding data packets with different ordered labels according to the ordered labels, and the data to be transmitted in each data packet has the orderliness. And, the obtained data packet is put into a message queue in the message queue means. When the number of the obtained data packets is multiple, the data packets are uniformly placed into each message queue, data to be sent are sent through the multiple message queues, and when the data migration equipment migrates the data of the message queues in the first machine room message queue device into the message queues of the second machine room message queue device, the data to be sent in the message queues are equivalently migrated, so that the orderliness among the data of the same ordered mark is ensured, and the message synchronization efficiency is improved. In addition, in the message queue device, if the number of the message queues changes, the number of the data packets put into each message queue changes correspondingly, and dynamic capacity expansion and reduction are realized.
As shown in fig. 5, an embodiment of the present application provides a data sending apparatus 300, which operates in a server in a cross-room data synchronization system, where the cross-room data synchronization system includes a message queue apparatus and a server; the message queue device comprises a plurality of message queues, the server is used for processing data to be sent and then putting the processed data into at least one of the message queues, and the message queues are first-in first-out queues used for sending the data. The server and the message queue device may be a server and a message queue device in a first machine room of the cross-machine room synchronization system. Specifically, the data transmission device 300 may include:
a data receiving module 310, configured to receive multiple data to be sent, where each data to be sent includes an ordered tag; a data dividing module 320, configured to divide the multiple data to be sent into one or more data packets according to the ordered labels in the data to be sent, where the data to be sent with the same ordered label is divided into the same data packet, and the data to be sent with different ordered labels is divided into different data packets; the enqueue module 330 is configured to place the one or more data packets into at least one of the plurality of message queues, where data to be sent in the same data packet is placed into the same message queue according to a receiving sequence.
Wherein the ordered label may be a routing field. The data dividing module 320 may be configured to arrange the data to be sent with the same ordered tag according to the receiving sequence to serve as a data packet.
If the data to be sent is divided into a plurality of data packets, the enqueue module 330 is configured to place the one or more data packets into at least one of the plurality of message queues, including: one or more data packets are placed into more than one message queue, respectively.
In one embodiment, the number of packets respectively put into the plurality of message queues by the enqueuing module 330 is not more than one.
Referring to fig. 6, based on the data transmission method and apparatus, the present application further provides a server 140, where the server 140 may serve as the server 111 or the server 123 of the system shown in fig. 1, may also serve as a physical device carrying the message queue apparatus 122 or the message queue apparatus 112, and may also serve as the data migration device 121 in fig. 1.
It should be noted that, when the message queue apparatus 112 or the message queue apparatus 122 is both installed on a device, the message queue apparatus 112 operates in a server 140, and the message queue apparatus 122 also operates in a server 140. When the message queue device 112 or the message queue device 122 is distributed in a plurality of physical devices, the message queue device 112 is distributed in a plurality of servers 140, and the message queue device 122 is also distributed in a plurality of servers 140.
The server 140 includes one or more processors 222 (only one of which is shown), memory 224, a network module 228, and a peripheral interface 230.
Those skilled in the art will appreciate that all other components are peripheral devices with respect to the processor 222, and the processor 222 is coupled to the peripheral devices through a plurality of peripheral interfaces 230. The peripheral interface 230 may be implemented based on the following standards: universal Asynchronous Receiver/Transmitter (UART), General Purpose Input/Output (GPIO), Serial Peripheral Interface (SPI), and Inter-Integrated Circuit (I2C), but the present invention is not limited to these standards. In some examples, the peripheral interface 230 may include only a bus. These controllers may also be separate from the peripheral interface 230 and integrated within the processor 222 or a corresponding peripheral.
The memory 224 may be used to store software programs and modules, and the processor 222 executes various functional applications and data processing by executing the software programs and modules stored in the memory 224. The memory 224 may include high speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid state memory. In some examples, the memory 224 may further include memory that is remotely located from the processor 222. In addition, the memory 224 may also store some data to be processed. For example, when message queue device 112 or message queue device 122 is carried by server 140, memory 224 may store data in a message queue.
The network module 228 is configured to receive and send network data to enable data interaction with other devices through a network.
The peripheral interface 230 is used for the server 140 to communicate with external devices in a wired manner. The peripheral interface can be a USB interface or an RS232 interface and the like.
Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
In the description herein, reference to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.
Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present application, "plurality" means at least two, e.g., two, three, etc., unless specifically limited otherwise.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and the scope of the preferred embodiments of the present application includes other implementations in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present application.
The logic and/or steps represented in the flowcharts or otherwise described herein, e.g., an ordered listing of executable instructions that can be considered to implement logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (server) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.
It should be understood that portions of the present application may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.
It will be understood by those skilled in the art that all or part of the steps carried by the method for implementing the above embodiments may be implemented by hardware related to instructions of a program, which may be stored in a computer readable storage medium, and when the program is executed, the program includes one or a combination of the steps of the method embodiments. For example, when the program is executed, the executed contents are the contents of step S110 to step S130 in the foregoing contents. In addition, functional units in the embodiments of the present application may be integrated into one processing module, or each unit may exist alone physically, or two or more units are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may also be stored in a computer readable storage medium.
The storage medium mentioned above may be a read-only memory, a magnetic or optical disk, etc. Although embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application, and that variations, modifications, substitutions and alterations may be made to the above embodiments by those of ordinary skill in the art within the scope of the present application.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not necessarily depart from the spirit and scope of the corresponding technical solutions in the embodiments of the present application.
Claims (7)
1. The data sending method is characterized by being applied to processing of payment services by a cross-machine-room data synchronization system, wherein the cross-machine-room data synchronization system comprises a message queue device and a server; the message queue device comprises a plurality of message queues, the server is used for processing data to be sent and then putting the processed data into at least one of the message queues, and the message queues are first-in first-out queues used for sending the data; the method comprises the following steps:
the server receives a plurality of data to be sent, wherein the data to be sent arrive at the server in different sequences on a link, order numbers in payment services are used as ordered marks, each data to be sent comprises an ordered mark, and more than one ordered mark corresponding to the data to be sent is included;
the server divides the data to be sent into a plurality of data packets according to the ordered marks in the data to be sent, wherein the data to be sent with the same ordered mark is divided into the same data packet, and the data to be sent with different ordered marks is divided into different data packets;
and the server puts the plurality of data packets into the plurality of message queues, wherein the data to be sent in the same data packet is put into the same message queue according to the sequence of the data to be sent received by the server, more than one data packet is put into each message queue, the number of the data packets put into each message queue is balanced, and the data to be sent is parallelly and uniformly sent through different message queues.
2. The method of claim 1, wherein segmenting the plurality of data to be transmitted into one or more data packets according to the ordered labels corresponding to the data to be transmitted comprises:
and arranging the data to be sent with the same ordered marks according to the receiving sequence to be used as a data packet.
3. The method of claim 1, wherein the step of placing the one or more data packets into at least one of the plurality of message queues includes not more than one difference between the numbers of data packets placed into the plurality of message queues, respectively.
4. A data sending device is characterized in that a server running in a cross-machine-room data synchronization system is used for processing payment services, and the cross-machine-room data synchronization system comprises a message queue device and a server; the message queue device comprises a plurality of message queues, the server is used for processing data to be sent and then putting the processed data into at least one of the message queues, and the message queues are first-in first-out queues used for sending the data; the data transmission apparatus includes:
the system comprises a data receiving module, a data sending module and a data sending module, wherein the data sending module is used for receiving a plurality of data to be sent, the data to be sent arrive at a server in different sequence on a link, order numbers in payment services are used as ordered marks, each data to be sent comprises an ordered mark, and more than one ordered mark corresponding to the data to be sent is included;
the data dividing module is used for dividing the data to be sent into a plurality of data packets according to the ordered marks in the data to be sent, wherein the data to be sent with the same ordered mark is divided into the same data packet, and the data to be sent with different ordered marks is divided into different data packets;
and the enqueuing module is used for putting the one or more data packets into the plurality of message queues, wherein the data to be sent in the same data packet is put into the same message queue according to the sequence of the server receiving the data to be sent, more than one data packet is put into each message queue, the number of the data packets put into each message queue is balanced, and the data to be sent is parallelly and uniformly sent through different message queues.
5. The apparatus of claim 4, wherein the data splitting module is configured to arrange the data to be transmitted with the same ordered label in the receiving order as a data packet.
6. A server, comprising one or more processors and memory;
one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs being executed by the processors to implement the method of any of claims 1-3.
7. A computer-readable storage medium storing program code executable by a processor, the program code being executable by the processor to implement the method of any one of claims 1 to 3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810069141.7A CN108347477B (en) | 2018-01-24 | 2018-01-24 | Data transmission method, device and server |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810069141.7A CN108347477B (en) | 2018-01-24 | 2018-01-24 | Data transmission method, device and server |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108347477A CN108347477A (en) | 2018-07-31 |
| CN108347477B true CN108347477B (en) | 2020-04-21 |
Family
ID=62960644
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810069141.7A Active CN108347477B (en) | 2018-01-24 | 2018-01-24 | Data transmission method, device and server |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108347477B (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110968433A (en) * | 2018-09-29 | 2020-04-07 | 北京京东尚科信息技术有限公司 | Information processing method and system and electronic equipment |
| CN109309729A (en) * | 2018-10-29 | 2019-02-05 | 杭州安恒信息技术股份有限公司 | The method and apparatus of data are transmitted under a kind of big flow faster |
| CN110636016B (en) * | 2019-08-08 | 2023-09-29 | 西安万像电子科技有限公司 | Data transmission method and system |
| CN111026813A (en) * | 2019-12-18 | 2020-04-17 | 紫光云(南京)数字技术有限公司 | High-availability quasi-real-time data synchronization method based on MySQL |
| CN111475315B (en) * | 2020-04-09 | 2024-04-16 | 广州方硅信息技术有限公司 | Server and subscription notification push control and execution method |
| CN112052104A (en) * | 2020-09-01 | 2020-12-08 | 掌阅科技股份有限公司 | Message queue management method based on multi-computer-room realization and electronic equipment |
| CN112068972A (en) * | 2020-09-01 | 2020-12-11 | 掌阅科技股份有限公司 | Message queue consumption method based on multi-computer room realization and electronic equipment |
| CN112468407A (en) * | 2020-12-15 | 2021-03-09 | 东莞中国科学院云计算产业技术创新与育成中心 | Data subpackage transmission method and device, computer equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2016010594A1 (en) * | 2014-07-18 | 2016-01-21 | Jive Communications, Inc. | Datacenter event stream processing in a network-based communication system |
| CN105471630A (en) * | 2015-11-18 | 2016-04-06 | 武汉日电光通信工业有限公司 | Northbound system message distribution system and northbound system message distribution method under large-capacity packet transmission system |
| CN107241281A (en) * | 2017-05-27 | 2017-10-10 | 上海东土远景工业科技有限公司 | A kind of data processing method and its device |
| CN107295036A (en) * | 2016-03-31 | 2017-10-24 | 华为技术有限公司 | A kind of data transmission method for uplink and data merge equipment |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103139732A (en) * | 2011-11-24 | 2013-06-05 | 方正国际软件有限公司 | Improved short message sending method and system |
| CN106204189A (en) * | 2016-06-24 | 2016-12-07 | 武汉合创源科技有限公司 | A kind of order processing method and system |
-
2018
- 2018-01-24 CN CN201810069141.7A patent/CN108347477B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2016010594A1 (en) * | 2014-07-18 | 2016-01-21 | Jive Communications, Inc. | Datacenter event stream processing in a network-based communication system |
| CN105471630A (en) * | 2015-11-18 | 2016-04-06 | 武汉日电光通信工业有限公司 | Northbound system message distribution system and northbound system message distribution method under large-capacity packet transmission system |
| CN107295036A (en) * | 2016-03-31 | 2017-10-24 | 华为技术有限公司 | A kind of data transmission method for uplink and data merge equipment |
| CN107241281A (en) * | 2017-05-27 | 2017-10-10 | 上海东土远景工业科技有限公司 | A kind of data processing method and its device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108347477A (en) | 2018-07-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108347477B (en) | Data transmission method, device and server | |
| CN108256002B (en) | Cross-machine-room data synchronization method, device, system and server | |
| US10331468B2 (en) | Techniques for routing service chain flow packets between virtual machines | |
| EP3028417A1 (en) | Data packet processing | |
| CN108347476B (en) | Cross-machine-room data synchronization method and device and server | |
| CN113485822A (en) | Memory management method, system, client, server and storage medium | |
| CN104104971A (en) | Video file processing method and system | |
| CN112398688A (en) | Container network configuration method, container network system, and storage medium | |
| CN104468401A (en) | Message processing method and device | |
| CN112822060B (en) | Host network construction method, device, system, medium and host | |
| CN111382115B (en) | Path creating method and device for network on chip and electronic equipment | |
| CN111246397B (en) | Cluster system, service access method, device and server | |
| CN112882792B (en) | Information loading method, computer device and storage medium | |
| CN105516264A (en) | Distributed cluster system based session sharing method, apparatus and system | |
| CN111092828B (en) | Network operation method, device, equipment and storage medium | |
| CN117633102A (en) | Block chain data integration method, device, computer equipment and storage medium | |
| CN109614354B (en) | Interface distribution method, board card, equipment and storage medium | |
| CN111679918B (en) | Message transmission method and device | |
| CN111262771B (en) | Virtual private cloud communication system, system configuration method and controller | |
| CN109656674B (en) | Computer equipment, virtualization chip and data transmission method | |
| CN107342981B (en) | Sensor data transmission method and device and virtual reality head-mounted equipment | |
| CN114897532A (en) | Operation log processing method, system, device, equipment and storage medium | |
| CN107210996B (en) | Service chain management method and device | |
| CN109862044B (en) | Conversion device, network equipment and data transmission method | |
| CN109510855B (en) | Event distribution system, method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: Changan town in Guangdong province Dongguan 523860 usha Beach Road No. 18 Applicant after: GUANGDONG OPPO MOBILE TELECOMMUNICATIONS CORP., Ltd. Address before: Changan town in Guangdong province Dongguan 523860 usha Beach Road No. 18 Applicant before: GUANGDONG OPPO MOBILE TELECOMMUNICATIONS CORP., Ltd. |
|
| CB02 | Change of applicant information | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |