CN110351370B - Data distribution method and device - Google Patents
Data distribution method and device Download PDFInfo
- Publication number
- CN110351370B CN110351370B CN201910631875.4A CN201910631875A CN110351370B CN 110351370 B CN110351370 B CN 110351370B CN 201910631875 A CN201910631875 A CN 201910631875A CN 110351370 B CN110351370 B CN 110351370B
- Authority
- CN
- China
- Prior art keywords
- data
- node
- permanent
- distribution
- end processor
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
Abstract
The invention relates to a data distribution method and a device, comprising the following steps: establishing a node tree with a plurality of permanent nodes, wherein at least one temporary node is arranged below each permanent node, and each temporary node corresponds to a terminal device for distribution; and each terminal device monitors the corresponding permanent node in real time, and if the terminal device monitors that the corresponding permanent node changes, the terminal device pulls the work order from the corresponding permanent node and delivers the work order according to the work order until the delivery is completed. The invention realizes the clustering deployment by correspondingly connecting at least one front-end processor under each permanent node, and each front-end processor monitors the corresponding permanent node in real time, so that the list information of all the front-end processors under the monitored permanent nodes can be obtained in each front-end processor and stored in the memory, and when a distribution task occurs, each front-end processor can balance the load by adding a load balancing algorithm according to the list information stored in the current memory, thereby realizing the high-efficiency distribution.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a data distribution method and apparatus.
Background
In the current data transaction distribution, a front-end processor is installed on a server of a supplier and a demander, then a transaction system pushes a corresponding work order to the front-end processor, and the front-end processor completes the data distribution according to the work order. Data distribution is an important ring in the data transaction link, and the data distribution device is complex and is a high-concurrency high-throughput device.
In the prior art, to realize high efficiency, stability and controllability in a huge data distribution device, the following problems are generally existed: (1) the server is down or the front-end processor fails to sense; (2) the existing front-end processor does not support cluster deployment, only can operate in a single machine, and the single machine is easy to generate performance bottleneck; (3) when a server is suddenly down, a network is in a problem or the service is unavailable, the dynamic sensing cannot be realized, and the automatic switching of a standby system is not available, so that the high availability of the device cannot be ensured.
Therefore, it is necessary to provide a data distribution method and device, which can realize real-time, efficient, stable and reliable data distribution to improve the overall distribution efficiency of the device.
Disclosure of Invention
The invention aims to provide a data distribution method and a data distribution device, which realize real-time, efficient, stable and reliable data distribution so as to improve the overall distribution efficiency of the device.
In order to solve the problems in the prior art, the invention provides a data distribution method, which comprises the following steps:
establishing a node tree with a plurality of permanent nodes, wherein at least one temporary node is arranged below each permanent node, and each temporary node corresponds to a terminal device for distribution;
and each terminal device monitors the corresponding permanent node in real time, if the terminal device monitors that the corresponding permanent node changes, the terminal device pulls a work order from the corresponding permanent node, and the work order is delivered according to the work order until the delivery is completed.
Optionally, in the data distribution method, the changing of the permanent node includes the following steps: and the permanent nodes of the data supplier and/or the data demander are modified according to the work order, and the work order information is stored in the permanent nodes of the data supplier and/or the data demander.
Optionally, in the data distribution method, each permanent node corresponds to one data supplier or one data demander, each temporary node corresponds to one terminal device for distribution, and the terminal device is a front-end processor.
Optionally, in the data distribution method, the forming of the permanent node includes the following steps: when the front-end processor of the data supplier and/or the data demander is started, registering basic information to a top node in the node tree;
the top node judges whether the data supplier and/or the data demander is registered for the first time or not according to the basic information, if so, a permanent node is newly built under the top node; otherwise, a temporary node is created under the existing permanent node of the data supplier and/or the data demander.
Optionally, in the data distribution method, the front-end processors corresponding to the temporary nodes under each permanent node form respective front-end processor clusters, and the distribution of the work orders includes the following steps:
and transmitting data from the data supplier front-end processor cluster to the data demander front-end processor cluster.
Optionally, in the data distribution method, during a period from generation of the work order to completion of distribution, the distribution task is monitored and alarmed, and the method includes the following steps:
and when monitoring that the permanent nodes of the data supplier and/or the data demander, the temporary nodes of the data supplier and/or the data demander and the distribution tasks are abnormal, alarming and executing a compensation mechanism.
Optionally, in the data distribution method, the compensation mechanism includes the following steps:
when the corresponding temporary node is disconnected due to the failure of the front-end processor or the network failure, the corresponding temporary node fails, and the work of the failure temporary node corresponding to the front-end processor is finished by the front-end processors corresponding to the other temporary nodes under the permanent node where the failure temporary node is located.
Optionally, in the data distribution method, while monitoring the distribution task in real time, the method further includes the following steps:
and archiving, backing up and analyzing the distribution log.
The invention also provides a data distribution device, comprising:
the central control unit is configured to comprise a node tree with a plurality of permanent nodes, at least one temporary node is arranged below each permanent node, and each temporary node corresponds to a terminal device for distribution; and each terminal device monitors the corresponding permanent node in real time, if the terminal device monitors that the corresponding permanent node changes, the terminal device pulls a work order from the corresponding permanent node, and the work order is delivered according to the work order until the delivery is completed.
Optionally, in the data distribution apparatus, the permanent node includes the following changes: and the permanent nodes of the data supplier and/or the data demander are modified according to the work order, and the work order information is stored in the permanent nodes of the data supplier and/or the data demander.
Optionally, in the data distribution apparatus, each permanent node corresponds to one data supplier or one data demander, each temporary node corresponds to one terminal device for distribution, and the terminal device is a front-end processor.
Optionally, in the data distribution apparatus, the central control unit is further configured to establish a permanent node, where the permanent node is established in the following manner: when the front-end processor of the data supplier and/or the data demander is started, registering basic information to a top node in the node tree;
the top node judges whether the data supplier and/or the data demander is registered for the first time or not according to the basic information, if so, a permanent node is newly built under the top node; otherwise, a temporary node is created under the existing permanent node of the data supplier and/or the data demander.
Optionally, in the data distribution device, the data distribution device further includes a front-end processor cluster unit, where the front-end processor cluster unit includes front-end processors corresponding to temporary nodes below the permanent nodes to form respective front-end processor clusters;
the work order distribution mode is as follows: and the central control unit controls the data supplier front-end processor cluster to transmit data to the data demander front-end processor cluster.
Optionally, in the data distribution device, the data distribution device further includes a distribution monitoring unit, and the distribution monitoring unit is configured to monitor and alarm the distribution tasks during a period from the generation of the work order to the completion of distribution, where the monitoring and alarm modes for the distribution tasks are as follows:
and when monitoring that the permanent nodes of the data supplier and/or the data demander, the temporary nodes of the data supplier and/or the data demander and the distribution task are abnormal, alarming and executing a compensation mechanism.
Optionally, in the data distribution apparatus, the compensation mechanism is as follows: when the corresponding temporary node is disconnected due to the failure of the front-end processor or the network failure, the corresponding temporary node fails, and the work of the failure temporary node corresponding to the front-end processor is finished by the front-end processors corresponding to the other temporary nodes under the permanent node where the failure temporary node is located.
Optionally, in the data distribution apparatus, the data distribution apparatus further includes a data analysis unit configured to perform archive backup and analysis on the distribution log while the distribution monitoring unit monitors the distribution tasks in real time.
In the data distribution method and device provided by the invention, at least one front-end processor is correspondingly connected under each permanent node to realize clustering deployment, and each front-end processor monitors the corresponding permanent node in real time, so that the list information of all the front-end processors under the monitored permanent nodes can be obtained in each front-end processor and stored in a memory, and when a distribution task occurs, each front-end processor can perform load balancing according to the list information stored in the current memory and a load balancing algorithm, thereby realizing high-efficiency distribution.
Drawings
Fig. 1 is a flowchart of a data distribution method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a single persistent node provided by an embodiment of the present invention;
fig. 3 is a schematic diagram of data distribution according to an embodiment of the present invention.
Detailed Description
The following describes in more detail embodiments of the present invention with reference to the schematic drawings. Advantages and features of the present invention will become apparent from the following description and claims. It is to be noted that the drawings are in a very simplified form and are not to precise scale, which is merely for the purpose of facilitating and distinctly claiming the embodiments of the present invention.
Hereinafter, if a method described herein includes a series of steps, and the order of such steps presented herein is not necessarily the only order in which such steps may be performed, some of the described steps may be omitted and/or some other steps not described herein may be added to the method.
The present invention provides a data distribution method, as shown in fig. 1, fig. 1 is a flowchart of a data distribution method provided in an embodiment of the present invention, where the data distribution method includes the following steps:
establishing a node tree with a plurality of permanent nodes, wherein at least one temporary node is arranged below each permanent node, and each temporary node corresponds to a terminal device for distribution;
and each terminal device monitors the corresponding permanent node in real time, if the terminal device monitors that the corresponding permanent node changes, the terminal device pulls a work order from the corresponding permanent node, and the work order is delivered according to the work order until the delivery is completed.
The invention realizes the clustering deployment by correspondingly connecting at least one front-end processor under each permanent node, and each front-end processor monitors the corresponding permanent node in real time, so that the list information of all the front-end processors under the monitored permanent nodes can be obtained in each front-end processor and stored in the memory, and when a distribution task occurs, each front-end processor can balance the load by adding a load balancing algorithm according to the list information stored in the current memory, thereby realizing the high-efficiency distribution.
Further, the permanent node change comprises the following steps: when the new order is put in storage, a work order is generated, and the central control unit automatically stores the work order information into the permanent nodes of the data supplier and/or the data demander, so that the permanent nodes of the data supplier and/or the data demander can be modified according to the work order.
As shown in fig. 2 and fig. 3, fig. 2 is a schematic diagram of a single permanent node provided in the embodiment of the present invention, and fig. 3 is a schematic diagram of data distribution provided in the embodiment of the present invention, where each permanent node corresponds to one data supplier or one data demander, each temporary node corresponds to one terminal device for distribution, and the terminal device is a front-end processor. In addition, the front-end processors corresponding to the temporary nodes under each permanent node form respective front-end processor clusters, and in fig. 3, a data supplier front-end processor cluster and a data demander front-end processor cluster are included, and each front-end processor cluster includes at least one front-end processor, for example, 1, 2, 3, 4, or 5 front-end processors may be included. Further, the distribution of the work orders comprises the following steps: and when the work order starts to be distributed, the data supplier needs to provide data to the data demander, and the data supplier front-end processor cluster transmits the data to the data demander front-end processor cluster. Specifically, for the cluster, a polling load balancing algorithm is generally used by default, the load balancing algorithm can be customized, the data supplier front-end processor calculates the front-end processor currently executing the distribution task according to the list information of the front-end processor cluster in the memory and the load balancing algorithm, and the corresponding front-end processor performs the data distribution task.
In the data distribution method, the node tree further has at least one top node, and the number of top nodes can be increased in the case that there are many permanent nodes in the node tree. Further, the forming of the permanent node comprises the following steps: when the front-end processor of the data supplier and/or the data demander is started, registering basic information to a top node in the node tree;
the top node judges whether the data supplier and/or the data demander is registered for the first time or not according to the basic information, if so, a permanent node is newly built under the top node; otherwise, a temporary node is created under the existing permanent node of the data supplier and/or the data demander.
Specifically, when a front-end processor of a data supplier or a data demander is started, basic information of the front-end processor is automatically registered to a top-level node of a central control unit, the central control unit maintains the node tree, if the data supplier or the data demander is registered for the first time, a permanent node containing the basic information of the data supplier or the data demander is created, a temporary node is created below the permanent node, the basic information of the current front-end processor is stored in the temporary node, and each front-end processor monitors a father node, namely, the permanent node; when starting other front-end processor, it will register the basic information of the front-end processor to the central control unit, at this moment, no longer creating permanent node, directly creating temporary node under the permanent node, said temporary node storing the basic information of the front-end processor, the front-end processor simultaneously monitoring father node, that is, monitoring permanent node. Furthermore, each data supplier or data demander deploys a corresponding number of front-end computers according to the actual situation, each data supplier or data demander corresponds to a permanent node, and stores the basic information of the data supplier or data demander on the permanent node, each front-end computer of the data supplier or data demander corresponds to each temporary node under the permanent node, and writes the information of ip address port and the like of each front-end computer into the corresponding temporary node; then monitoring the permanent node of the data supplier or the data demander, and finishing the service registration and monitoring of the data supplier or the data demander front-end processor.
Preferably, after the front-end processor pulls the work order from the corresponding permanent node thereof, before the front-end processor delivers the work order, the method may further include the following steps; checking delivery conditions of temporary nodes of the data supplier and the data demander. Specifically, the central control unit controls the data demander front-end processor to initialize and check the self system condition, if the distribution condition is met, the distribution monitoring unit feeds back the condition that the data demander front-end processor is monitored to the central control unit, the central control unit controls the data supplier front-end processor to initialize similarly, and if the distribution condition is met, the distribution monitoring unit feeds back the condition that the data supplier front-end processor is monitored to the central control unit, so that the distribution condition is confirmed.
Further, during the period from the generation of the work order to the completion of the delivery, the monitoring and the alarming of the delivery tasks may specifically include the following steps:
and when monitoring that the permanent nodes of the data supplier and/or the data demander, the temporary nodes of the data supplier and/or the data demander and the distribution tasks are abnormal, alarming and executing a compensation mechanism. The invention can monitor the execution condition of the distribution tasks in real time, and can automatically warn if the distribution is abnormal, stop the continuous execution of the tasks and inform the central control unit to execute a compensation mechanism.
In one embodiment, the compensation mechanism may include a manner that, since each front-end processor of the data supplier or the data demander corresponds to a temporary node created under the permanent node, once a certain front-end processor or network fails to cause disconnection of the corresponding temporary node, the central control unit is no longer connected, and the temporary node is automatically deleted due to the characteristics of the temporary node. Other front-end computers monitor the permanent node, so once the temporary node is lost under the permanent node, the other front-end computers can sense the change and update the list information of the front-end computer cluster in the memory, and meanwhile, the central control unit can also sense the failure and the loss of the connection of a certain front-end computer of the current data supplier or the data demander. Therefore, if an exception occurs in the delivery process or the delivery process exceeds the threshold value, the delivery monitoring unit can perform early warning and notify the central control unit to execute a corresponding compensation mechanism, for example, the work of the failure temporary node corresponding to the front-end processor is completed by the other temporary nodes under the permanent node where the failure temporary node is located corresponding to the front-end processor, and a technician can be notified to perform manual intervention under the condition that the repeated retry compensation fails, so that the delivery task can be successfully completed.
In the data distribution method, the method of generating the work order includes: and creating a standardized data product order, and establishing the work order according to the order. Specifically, after the new order is put in storage, a work order is generated, the central control unit automatically modifies the permanent nodes of the data supply and demand party according to the work order, and after the data supply and demand party front-end processor monitors the change of the permanent nodes, the work order is pulled to the central control unit, and then the distribution task is completed according to the work order. In one embodiment, the method for creating the work order according to the order comprises the following steps: the order comprises a supplier number, an acquirer number, a product name, a product type, a pricing mode, a unit price, a distribution mode and the like; the work order is extracted in one layer based on the order, and the work order is mainly responsible for distribution, so the work order mainly comprises the address of a front-end processor of a data supplier, the address of a front-end processor of a data demander, a distribution mode, distribution detailed information and the like. And after acquiring the work order, the data supplier front-end processor establishes connection according to the address of the data demander front-end processor in the work order, and then distributes according to the distribution mode and the distribution detailed information.
Further, when the distribution tasks are monitored in real time, the method further comprises the following steps: and archiving, backing up and analyzing the distribution log. For example, one may focus on analyzing a distribution log generated by the data distribution facility to aggregate the distribution volume for clearing.
The present invention also provides a data delivery apparatus, as shown in fig. 2 and 3, including:
the central control unit is configured to comprise a node tree with a plurality of permanent nodes, at least one temporary node is arranged below each permanent node, and each temporary node corresponds to a terminal device for distribution; and each terminal device monitors the corresponding permanent node in real time, if the terminal device monitors that the corresponding permanent node changes, the terminal device pulls a work order from the corresponding permanent node, and the work order is delivered according to the work order until the delivery is completed.
The invention realizes the clustering deployment by correspondingly connecting at least one front-end processor under each permanent node, and each front-end processor monitors the corresponding permanent node in real time, so that the list information of all the front-end processors under the monitored permanent nodes can be obtained in each front-end processor and stored in the memory, and when a distribution task occurs, each front-end processor can balance the load by adding a load balancing algorithm according to the list information stored in the current memory, thereby realizing the high-efficiency distribution.
Further, the permanent nodes include the following variations: when the new order is put in storage, a work order is generated, and the central control unit automatically stores the work order information into the permanent nodes of the data supplier and/or the data demander, so that the permanent nodes of the data supplier and/or the data demander can be modified according to the work order.
In the data distribution device, each permanent node corresponds to one data supplier or one data demander, each temporary node corresponds to one terminal device for distribution, and the terminal device is a front-end processor. Further, the data distribution device further includes a front-end processor cluster unit, where the front-end processor cluster unit includes front-end processors corresponding to the temporary nodes below the permanent nodes to form respective front-end processor clusters, and in fig. 3, the front-end processor cluster includes a data supplier front-end processor cluster and a data supplier front-end processor cluster, and each front-end processor cluster includes at least one front-end processor, for example, 1, 2, 3, 4, or 5 front-end processors may be included. The work order distribution mode is as follows: and when the work order starts to be distributed, the data supplier needs to provide data to the data demander, and the data supplier front-end processor cluster transmits the data to the data demander front-end processor cluster. Specifically, for the cluster, a polling load balancing algorithm is generally used by default, the load balancing algorithm can be customized, the data supplier front-end processor calculates the front-end processor currently executing the distribution task according to the list information of the front-end processor cluster in the memory and the load balancing algorithm, and the corresponding front-end processor performs the data distribution task.
Furthermore, the node tree of the central control unit also has at least one top node, and the number of the top nodes can be increased under the condition that the number of the permanent nodes of the node tree is large. The central control unit is further configured to establish a permanent node in the following manner: when the front-end processor of the data supplier and/or the data demander is started, registering basic information to the top-level node;
the top node judges whether the data supplier and/or the data demander is registered for the first time or not according to the basic information, if so, a permanent node is newly built under the top node; otherwise, a temporary node is created under the existing permanent node of the data supplier and/or the data demander.
Specifically, when a front-end processor of a data supplier or a data demander is started, the front-end processor automatically registers its own basic information with a top-level node of a central control unit, the central control unit maintains the node tree, if the data supplier or the data demander registers for the first time, a permanent node containing the basic information of the data supplier or the data demander is created, then a temporary node is created under the permanent node, the basic information of the current front-end processor is stored in the temporary node, and each front-end processor monitors its parent node, i.e. monitors the permanent node; when other front-end computers are started, the basic information of the front-end computers is also registered to the central control unit, at this time, the permanent nodes are not created any more, the temporary nodes are directly created under the permanent nodes, the temporary nodes store the basic information of the corresponding front-end computers, and the front-end computers monitor father nodes, namely the permanent nodes. Furthermore, each data supplier or data demander deploys a corresponding number of front-end computers according to the actual situation, each data supplier or data demander corresponds to a permanent node, and stores the basic information of the data supplier or data demander on the permanent node, each front-end computer of the data supplier or data demander corresponds to each temporary node under the permanent node, and writes the information of ip address port and the like of each front-end computer into the corresponding temporary node; then monitoring the permanent node of the data supplier or the data demander, and finishing the service registration and monitoring of the data supplier or the data demander front-end processor.
Preferably, the central control unit may further check delivery conditions of front-end computers of the data supplier and the data demander, and the check of the delivery conditions may be performed before the front-end computers pull the work orders from their corresponding permanent nodes and deliver the work orders according to the work orders. The specific manner of checking the distribution conditions is as follows: the central control unit controls the data demander front-end processor to initialize and check the system condition of the data demander front-end processor, if the distribution condition is met, the distribution monitoring unit feeds back the condition that the data demander front-end processor is monitored to be ready to the central control unit, the central control unit controls the data supplier front-end processor to initialize the same, and if the distribution condition is met, the distribution monitoring unit feeds back the condition that the data supplier front-end processor is monitored to be ready to the central control unit, so that the distribution condition is confirmed.
Further, the data distribution device further includes a distribution monitoring unit, and the distribution monitoring unit is configured to monitor and alarm the distribution tasks during the period from the generation of the work order to the completion of the distribution, and the monitoring and alarm modes of the distribution tasks are as follows:
and when monitoring that the permanent nodes of the data supplier and/or the data demander, the temporary nodes of the data supplier and/or the data demander and the distribution tasks are abnormal, alarming and executing a compensation mechanism. The invention can monitor the execution condition of the delivery tasks in real time, and if abnormal delivery can be automatically warned, the invention stops the task to be continuously executed and informs the central control unit to execute a compensation mechanism.
In one embodiment, the compensation mechanism may include a manner that, since each front-end processor of the data supplier or the data demander corresponds to a temporary node created under the permanent node, once a certain front-end processor or network fails to cause disconnection of the corresponding temporary node, the central control unit is no longer connected, and the temporary node is automatically deleted due to the characteristics of the temporary node. Other front-end computers monitor the permanent node, so once the temporary node is lost under the permanent node, the other front-end computers can sense the change and update the list information of the front-end computer cluster in the memory, and meanwhile, the central control unit can also sense the failure and the loss of the connection of a certain front-end computer of the current data supplier or the data demander. Therefore, if an abnormality occurs in the distribution process or the distribution process exceeds a threshold value, the distribution monitoring unit can give an early warning to notify the central control unit to execute a corresponding compensation mechanism, for example, the work of the failure temporary node corresponding to the front-end processor is completed by the other temporary nodes under the permanent node where the failure temporary node is located corresponding to the front-end processor, and a technician can be notified to perform manual intervention under the condition that the repeated retry compensation fails, so that the distribution task can be successfully completed.
Preferably, the data distribution apparatus further includes a portal unit configured to provide an interactive interface to the data supplier and the data demander, create a standardized data product order in the portal unit, and the central control unit creates the work order according to the order. Specifically, after the new order is put in storage, a work order is generated, the central control unit automatically modifies the permanent nodes of the data supply and demand party according to the work order, and after the data supply and demand party front-end processor monitors the change of the permanent nodes, the work order is pulled to the central control unit, and then the distribution task is completed according to the work order. In one embodiment, the method for creating the work order according to the order comprises the following steps: the order comprises a supplier number, an acquirer number, a product name, a product type, a pricing mode, a unit price, a distribution mode and the like; the work order is extracted in one layer based on the order, and the work order is mainly responsible for distribution, so the work order mainly comprises the address of a front-end processor of a data supplier, the address of a front-end processor of a data demander, a distribution mode, distribution detailed information and the like. And after acquiring the work order, the data supplier front-end processor establishes connection according to the address of the data demander front-end processor in the work order, and then distributes according to the distribution mode and the distribution detailed information.
Preferably, the data distribution device further includes a data analysis unit configured to perform archive backup and analysis on the distribution log while monitoring the distribution tasks in real time. For example, one may focus on analyzing a distribution log generated by the data distribution facility to aggregate the distribution volume for clearing.
The data distribution device provided by the invention comprises the following unit modules:
portal unit: a portal for providing data transaction services to members. Through the portal unit, the members of the data supplier and the data demander can realize the services of commodity listing, demand release, data ordering, transaction management, information inquiry, clearing and the like of data transaction.
A data analysis unit: the system is used for analyzing the distribution logs generated by the data distribution device and summarizing the distribution amount for the portal unit to clear and settle.
A data supplier: after the portal unit opens an account, a transaction item interconnection object may be created at the portal unit and listed.
A data demander: after the portal unit opens an account, the data supplier may be purchased at the portal unit to hang up the shelved transaction interconnect object.
A central control unit: the system is used for managing and controlling all supply and demand party front-end computers and distribution tasks of the data trading center. The method can manage the front-end computers of the supply and demand parties in the central control unit, dynamically monitor the running state of each front-end computer, issue the tasks to each front-end computer when the tasks are distributed, and execute a compensation mechanism when the distribution is abnormal.
A delivery monitoring unit: the distribution monitoring unit can check the distribution conditions of all the front-end computers and is responsible for monitoring, early warning and termination of distribution tasks and statistical reporting of the distribution conditions.
Permanent node: after a node of a node tree in the central control unit is created, the unregistered data supplier front-end processor and the unregistered data supplier front-end processor can be connected with the central control unit through the node.
A permanent node: after a node of a node tree in the central control unit is created, if the node is not actively triggered to be deleted, the node cannot be deleted and always exists on the central control unit.
And (4) temporary nodes: and a node of a node tree in the central control unit, wherein the life cycle of the node is bound with the client session, and the node is automatically cleared when the client session fails.
In summary, in the data distribution method and apparatus provided by the present invention, at least one front-end processor is correspondingly connected to each permanent node to implement clustered deployment, and each front-end processor monitors the corresponding permanent node in real time, so that each front-end processor can obtain list information of all front-end processors under the monitored permanent nodes and store the list information in a memory, and when a distribution task occurs, each front-end processor can perform load balancing according to the list information stored in the current memory and a load balancing algorithm, thereby implementing high-efficiency distribution; the invention also monitors the permanent nodes of the data supplier and/or the data demander, the temporary nodes of the data supplier and/or the data demander and the distribution tasks during the period from the generation of the work order to the completion of the distribution, thereby realizing the real-time monitoring of the faults in the distribution and the timely processing and alarming.
The above description is only a preferred embodiment of the present invention, and does not limit the present invention in any way. It will be understood by those skilled in the art that various changes, substitutions and alterations can be made herein without departing from the spirit and scope of the invention as defined by the appended claims.
Claims (10)
1. A data distribution method, comprising the steps of:
establishing a node tree with a plurality of permanent nodes, wherein each permanent node corresponds to a data supplier or a data demander, each permanent node is provided with at least one temporary node below, each temporary node corresponds to a terminal device for distribution, and the formation of the permanent nodes comprises the following steps: when the front-end processor of the data supplier and/or the data demander is started, registering basic information to a top-level node in the node tree, and judging whether the data supplier and/or the data demander is registered for the first time by the top-level node according to the basic information, if so, newly building a permanent node under the top-level node, otherwise, building a temporary node under the existing permanent node of the data supplier and/or the data demander;
each terminal device monitors the corresponding permanent node in real time, if the terminal device monitors that the corresponding permanent node changes, the terminal device pulls a work order from the corresponding permanent node and delivers according to the work order until the delivery is completed,
front-end processors corresponding to the temporary nodes under each permanent node form respective front-end processor clusters, and the distribution of the work orders comprises the following steps: the data supplier front-end processor cluster transmits data to the data demander front-end processor cluster, concretely, the data supplier front-end processor calculates the front-end processor executing the current distribution task according to the list information of the front-end processor cluster in the memory and the load balancing algorithm, the corresponding front-end processor carries out the data distribution task,
the permanent node change comprises the following steps: and the permanent nodes of the data supplier and/or the data demander are modified according to the work order, and the work order information is stored in the permanent nodes of the data supplier and/or the data demander.
2. A data distribution method as claimed in claim 1, wherein said terminal device is a front-end processor.
3. A data distribution method as claimed in claim 1 wherein said distribution tasks are monitored and alerted during the period from the generation of said work order until the completion of the distribution, comprising the steps of:
and when monitoring that the permanent nodes of the data supplier and/or the data demander, the temporary nodes of the data supplier and/or the data demander and the distribution tasks are abnormal, alarming and executing a compensation mechanism.
4. A data distribution method as claimed in claim 3, wherein said compensation mechanism comprises the steps of:
when the corresponding temporary node is disconnected due to the failure of the front-end processor or the network failure, the corresponding temporary node fails, and the work of the failure temporary node corresponding to the front-end processor is finished by the front-end processors corresponding to the other temporary nodes under the permanent node where the failure temporary node is located.
5. A data distribution method according to claim 1, wherein while monitoring the distribution task in real time, further comprising the steps of:
and archiving, backing up and analyzing the distribution log.
6. A data distribution apparatus, comprising:
the central control unit is configured to comprise a node tree with a plurality of permanent nodes, each permanent node corresponds to a data supplier or a data demander, at least one temporary node is arranged below each permanent node, and each temporary node corresponds to a terminal device for distribution; each terminal device monitors the corresponding permanent node in real time, if the terminal device monitors that the corresponding permanent node changes, the terminal device pulls a work order from the corresponding permanent node and delivers according to the work order until the delivery is completed,
the central control unit is further configured to establish a permanent node in the following manner: when the front-end processor of the data supplier and/or the data demander is started, registering basic information to a top node in the node tree; the top node judges whether the data supplier and/or the data demander is registered for the first time or not according to the basic information, if so, a permanent node is newly built under the top node; otherwise, a temporary node is created under the existing permanent node of the data supplier and/or the data demander,
the data distribution device also comprises a front-end processor cluster unit, wherein the front-end processor cluster unit comprises front-end processors corresponding to the temporary nodes under the permanent nodes to form respective front-end processor clusters; the work order distribution mode is as follows: the central control unit controls the data supplier front-end processor cluster to transmit data to the data demander front-end processor cluster, concretely, the data supplier front-end processor calculates the front-end processor currently executing the distribution task according to the list information of the front-end processor cluster in the memory and the load balancing algorithm, the corresponding front-end processor carries out the data distribution task,
the permanent nodes include the following changes: and the permanent nodes of the data supplier and/or the data demander are modified according to the work order, and the work order information is stored in the permanent nodes of the data supplier and/or the data demander.
7. The data distribution apparatus of claim 6, wherein the terminal device is a front-end processor.
8. The data distribution apparatus according to claim 6, wherein the data distribution apparatus further comprises a distribution monitoring unit configured to monitor and alarm the distribution tasks during the period from the generation of the work order until the completion of the distribution, in the following manner:
and when monitoring that the permanent nodes of the data supplier and/or the data demander, the temporary nodes of the data supplier and/or the data demander and the distribution tasks are abnormal, alarming and executing a compensation mechanism.
9. The data distribution apparatus of claim 8, wherein the compensation mechanism is as follows: when the corresponding temporary node is disconnected due to the failure of the front-end processor or the network failure, the corresponding temporary node fails, and the work of the failure temporary node corresponding to the front-end processor is finished by the front-end processors corresponding to the other temporary nodes under the permanent node where the failure temporary node is located.
10. The data distribution apparatus according to claim 6, wherein the data distribution apparatus further comprises a data analysis unit configured to archive and analyze the distribution log while the distribution monitoring unit monitors the distribution tasks in real time.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910333143 | 2019-04-24 | ||
| CN2019103331437 | 2019-04-24 |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110351370A CN110351370A (en) | 2019-10-18 |
| CN110351370B true CN110351370B (en) | 2022-06-21 |
Family
ID=68176151
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910631875.4A Active CN110351370B (en) | 2019-04-24 | 2019-07-12 | Data distribution method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110351370B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111898947A (en) * | 2020-07-21 | 2020-11-06 | 北京京东振世信息技术有限公司 | Method, device, equipment and computer readable medium for monitoring goods distribution task |
| CN115361439B (en) * | 2022-07-12 | 2024-03-15 | 北京奇艺世纪科技有限公司 | Node management method, node management device, electronic equipment and storage medium |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104486108A (en) * | 2014-12-08 | 2015-04-01 | 畅捷通信息技术股份有限公司 | Node configuration method base on Zookeeper and node configuration system based on Zookeeper |
| CN104504547A (en) * | 2014-12-15 | 2015-04-08 | 北京京东尚科信息技术有限公司 | Work order generating method and system supporting goods physical distribution |
| CN107707586A (en) * | 2016-08-08 | 2018-02-16 | 北京京东尚科信息技术有限公司 | Load-balancing method and device |
| CN107872539A (en) * | 2017-12-15 | 2018-04-03 | 安徽长泰信息安全服务有限公司 | A kind of data handling system and method based on cloud computing platform |
| CN107948017A (en) * | 2017-09-30 | 2018-04-20 | 用友金融信息技术股份有限公司 | Highly available cluster system |
| CN108366086A (en) * | 2017-12-25 | 2018-08-03 | 聚好看科技股份有限公司 | A kind of method and device of control business processing |
| CN108763389A (en) * | 2018-05-18 | 2018-11-06 | 上海数据交易中心有限公司 | The integration method and device of data, storage medium, terminal |
| CN109639480A (en) * | 2018-12-10 | 2019-04-16 | 北京京东金融科技控股有限公司 | A kind of system and method that data report |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| SE0200418D0 (en) * | 2002-02-13 | 2002-02-13 | Ericsson Telefon Ab L M | A method and apparatus for computer load sharing and data distribution |
| US7206836B2 (en) * | 2002-09-23 | 2007-04-17 | Sun Microsystems, Inc. | System and method for reforming a distributed data system cluster after temporary node failures or restarts |
| CN104050553A (en) * | 2014-06-12 | 2014-09-17 | 长春理工大学 | Inventory and distribution management system based on cloud computing |
| US9619210B2 (en) * | 2015-05-14 | 2017-04-11 | Walleye Software, LLC | Parsing and compiling data system queries |
-
2019
- 2019-07-12 CN CN201910631875.4A patent/CN110351370B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104486108A (en) * | 2014-12-08 | 2015-04-01 | 畅捷通信息技术股份有限公司 | Node configuration method base on Zookeeper and node configuration system based on Zookeeper |
| CN104504547A (en) * | 2014-12-15 | 2015-04-08 | 北京京东尚科信息技术有限公司 | Work order generating method and system supporting goods physical distribution |
| CN107707586A (en) * | 2016-08-08 | 2018-02-16 | 北京京东尚科信息技术有限公司 | Load-balancing method and device |
| CN107948017A (en) * | 2017-09-30 | 2018-04-20 | 用友金融信息技术股份有限公司 | Highly available cluster system |
| CN107872539A (en) * | 2017-12-15 | 2018-04-03 | 安徽长泰信息安全服务有限公司 | A kind of data handling system and method based on cloud computing platform |
| CN108366086A (en) * | 2017-12-25 | 2018-08-03 | 聚好看科技股份有限公司 | A kind of method and device of control business processing |
| CN108763389A (en) * | 2018-05-18 | 2018-11-06 | 上海数据交易中心有限公司 | The integration method and device of data, storage medium, terminal |
| CN109639480A (en) * | 2018-12-10 | 2019-04-16 | 北京京东金融科技控股有限公司 | A kind of system and method that data report |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110351370A (en) | 2019-10-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20230308525A1 (en) | Embedded database as a microservice for distributed container cloud platform | |
| KR101888029B1 (en) | Method and system for monitoring virtual machine cluster | |
| CN105245373B (en) | A kind of container cloud platform system is built and operation method | |
| EP1650653B1 (en) | Remote enterprise management of high availability systems | |
| US6138249A (en) | Method and apparatus for monitoring computer systems during manufacturing, testing and in the field | |
| CN105556499B (en) | Intelligent auto zoom | |
| CN108874567B (en) | Service processing method and system | |
| CN110351370B (en) | Data distribution method and device | |
| CN104769919A (en) | Load balancing access to replicated databases | |
| CN111552556B (en) | GPU cluster service management system and method | |
| CN107197012B (en) | Service publishing and monitoring system and method based on metadata management system | |
| CN107660289A (en) | Automatic network controls | |
| CN106156939A (en) | Dispatching System based on job stream and application process | |
| CN111857998A (en) | Configurable timing task scheduling method and system | |
| CN117130730A (en) | Metadata management method for federal Kubernetes cluster | |
| CN114253748B (en) | Message processing system and message processing method | |
| CN110798339A (en) | Task disaster tolerance method based on distributed task scheduling framework | |
| CN109995585B (en) | Exception handling method, device and storage medium | |
| CN108769170A (en) | A kind of cluster network fault self-checking system and method | |
| CN108737543B (en) | Distributed Internet of things middleware and working method | |
| CN109639755B (en) | Associated system server decoupling method, device, medium and electronic equipment | |
| CN105897487B (en) | Equipment management method and device for operation and maintenance system | |
| CN110275795A (en) | A kind of O&M method and device based on alarm | |
| CN105025179A (en) | Method and system for monitoring service agents of call center | |
| CN115273354A (en) | Bank self-service equipment management method and system and computer equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |