CN112953757B - Data distribution method, system and computer equipment - Google Patents
Data distribution method, system and computer equipment Download PDFInfo
- Publication number
- CN112953757B CN112953757B CN202110105058.2A CN202110105058A CN112953757B CN 112953757 B CN112953757 B CN 112953757B CN 202110105058 A CN202110105058 A CN 202110105058A CN 112953757 B CN112953757 B CN 112953757B
- Authority
- CN
- China
- Prior art keywords
- data
- distribution
- module
- message queue
- distributor
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000009826 distribution Methods 0.000 title claims abstract description 207
- 238000000034 method Methods 0.000 title claims abstract description 43
- 238000013506 data mapping Methods 0.000 claims abstract description 44
- 238000007726 management method Methods 0.000 claims abstract description 35
- 238000013499 data model Methods 0.000 claims abstract description 30
- 238000004891 communication Methods 0.000 claims description 23
- 239000012634 fragment Substances 0.000 claims description 15
- 230000008569 process Effects 0.000 claims description 10
- 238000000605 extraction Methods 0.000 claims description 6
- 238000013507 mapping Methods 0.000 claims description 5
- 239000000284 extract Substances 0.000 claims description 4
- 230000005540 biological transmission Effects 0.000 abstract description 14
- 238000010586 diagram Methods 0.000 description 6
- 230000008901 benefit Effects 0.000 description 5
- 238000004590 computer program Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 5
- 238000012545 processing Methods 0.000 description 5
- 238000003860 storage Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 4
- 238000004519 manufacturing process Methods 0.000 description 3
- 239000002699 waste material Substances 0.000 description 3
- 230000008878 coupling Effects 0.000 description 2
- 238000010168 coupling process Methods 0.000 description 2
- 238000005859 coupling reaction Methods 0.000 description 2
- 230000002708 enhancing effect Effects 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 108010001267 Protein Subunits Proteins 0.000 description 1
- 230000005856 abnormality Effects 0.000 description 1
- 230000001133 acceleration Effects 0.000 description 1
- 238000005520 cutting process Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 238000009776 industrial production Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 238000010223 real-time analysis Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/0803—Configuration setting
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0654—Management of faults, events, alarms or notifications using network fault recovery
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The application relates to a data distribution method, a system and computer equipment, wherein the data distribution method comprises the following steps: a parameter configuration step, setting parameters of a distribution management device and starting a data distribution device, wherein the distribution management device generates a consumer of an incoming message queue, a producer of an outgoing message queue and a corresponding data mapping table according to the parameters; a data inflow step, wherein the distribution management device matches the streaming data source with a data model table and obtains each field of the streaming data source, and the streaming data source enters the data distribution device through an incoming message queue according to a consumer; and a data distribution step, wherein the data distribution device selects corresponding fields in the streaming data source according to the data mapping table and sends the corresponding fields to the corresponding data service modules according to the producer. According to the method and the device, only the needed data dimension is written into the message queue, so that resources consumed by redundant data in network transmission are reduced, and the data distribution efficiency is improved.
Description
Technical Field
The present disclosure relates to the field of data distribution technologies, and in particular, to a data distribution method, system, and computer device.
Background
In industrial production, key maintenance objects comprise equipment, and because the abnormality of the equipment directly affects the production result, the real-time performance requirement of equipment data is higher, foreground business application needs to rely on the equipment data to perform state and early warning real-time monitoring and management, the method of manually copying and pasting data files between machines is difficult to keep up with the requirement of real-time analysis of the data, and a real-time data transmission system is needed to connect the equipment with data service.
The prior art scheme is based on a data distribution system of publish-subscribe: the publish-subscribe data distribution system is the mainstream data distribution system model. In a publish-subscribe based data distribution system, a data source producer publishes data to a topic, a consumer pulls topic data or a topic pushes data to a consumer. The data published by the producer can be obtained by all consumers. The above prior art has the following disadvantages: in a publish-subscribe based data distribution system, all consumers acquire the full amount of data once the producer publishes the data. In an industrial equipment scenario, the data of the equipment contains a large number of fields, and consumers often only need a certain number of fields in one dimension, resulting in a large waste of network transmission resources. Meanwhile, the data distribution system based on the publish-subscribe has lower fault tolerance performance in a push mode, and the performance of the whole system can be influenced by single consumer faults; in the pull mode, multiple consumers pull data at the same time, which can instantly pull up network I/O of the system, and cannot effectively cope with peaks and valleys of the data stream.
At present, no effective solution is proposed for the problem of network transmission resource waste caused in the related technology.
Disclosure of Invention
The embodiment of the application provides a data distribution method, a data distribution system and computer equipment, which are used for at least solving the problem of network transmission resource waste in the related technology.
In a first aspect, an embodiment of the present application provides a data distribution method, which is applied to a data distribution system, including the following steps:
a parameter configuration step, setting a parameter of a distribution management device and starting a data distribution device, wherein the distribution management device generates a consumer of an incoming message queue, a producer of an outgoing message queue and a corresponding data mapping table according to the parameter;
a data inflow step, wherein the distribution management device matches a streaming data source with a data model table and obtains each field of the streaming data source, and the streaming data source enters the data distribution device through the incoming message queue according to the consumer;
and a data distribution step, wherein the data distribution device selects the corresponding field in the stream data source according to the data mapping table, and sends the corresponding field to a corresponding data service module according to the producer.
In some embodiments, the parameters in the parameter configuration step include a data model of the in-message queue, out-message queue metadata, and a mapping manner of each data mapping module.
In some embodiments, the data streaming step specifically includes:
the streaming data source enters a distributor of the data distribution device through the fragments of the incoming message queue, and the distributor corresponds to the fragments one by one.
In some embodiments, the data distribution step specifically includes the following steps:
a data receiving step, namely pushing the streaming data source to a corresponding number of distribution modules according to a preset distribution target number;
a field extraction step, wherein a data mapping module of each distribution module performs field extraction on the streaming data source according to the data mapping table;
and a data sending step, wherein the extracted fields are sent to the data service module through the message output queue according to the producer.
In some embodiments, the connection between the distributor and the distribution to distribution modules comprises:
in a distributed environment, the distributor and the distribution module communicate through a TCP protocol;
in a stand-alone environment, asynchronous communication between the dispatcher and the dispatch module is performed through an observer mode.
In some embodiments, the distributor and the distribution module record a state once every time a set time passes, and the distributor and the distribution module write the sent field into the same cache area while sending the field downstream;
in a distributed environment, after a plurality of retry failures, the distributor distributes and starts a new distribution unit by the distribution management device, and the distributor resends the data in the buffer to the new distribution unit.
In a second aspect, an embodiment of the present application provides a data distribution system, applying the data distribution method in the first aspect, including a distribution management device, a data distribution device, a message in queue module, and a message out queue module, where:
the distribution management device stores a data model table, a data mapping table and outgoing message queue metadata;
the data distribution device comprises a distributor and a distribution module;
the distribution management device is matched with a streaming data source according to the data model table, and the streaming data source enters the distributor through the message-entering queue module;
the distribution module extracts the streaming data source according to the data mapping table, and sends the extracted streaming data source to a data service module through the message-out queue module according to the message-out queue metadata.
In some of these embodiments, the dispenser and the dispensing unit are provided as one or more;
the distributor corresponds to the fragments in the message queue module one by one, and the streaming data sources in the fragments are correspondingly transmitted to the distributor;
the number of the distribution modules is adapted to the preset distribution target number, the distribution modules process the streaming data sources according to the corresponding distribution targets, and the processed streaming data sources are transmitted to the corresponding data service modules.
In some of these embodiments, the distribution module comprises a data mapping unit and a data sending unit, wherein:
the data mapping unit converts the data model of the streaming data source into a data model of an outgoing message queue according to the data mapping table, and sends the processed streaming data source to the corresponding data sending unit;
and the data sending unit sends the streaming data source to the corresponding message outlet queue module.
In some of these embodiments, it comprises:
in a distributed environment, a communication unit is arranged between the distributor and the distribution module, and the distributor and the distribution module are communicated through a TCP protocol;
in a stand-alone environment, the observer mode is used for asynchronous communication.
Compared with the related art, the data distribution method, the system and the computer equipment provided by the embodiment of the application have the advantages that on one hand, only the needed data dimension is written into the message queue, so that the resources consumed by redundant data in network transmission are reduced, and the data distribution efficiency is improved; on the other hand, the data distribution module distributes data to the message queue instead of directly to the consumer, and can disperse a part of pressure to the message queue when the data service consumer generates back pressure, thereby enhancing the stability of the system.
The details of one or more embodiments of the application are set forth in the accompanying drawings and the description below to provide a more thorough understanding of the other features, objects, and advantages of the application.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute an undue limitation to the application. In the drawings:
FIG. 1 is a flow chart of a data distribution method according to an embodiment of the present application;
FIG. 2 is a flow chart of data distribution steps according to an embodiment of the present application;
FIG. 3 is a flow chart of a data distribution method according to a preferred embodiment of the present application;
FIG. 4 is a block diagram of a data distribution system according to an embodiment of the present application;
FIG. 5 is a block diagram of a preferred architecture of a data distribution system according to an embodiment of the present application;
fig. 6 is a schematic diagram of a hardware structure of a computer device according to an embodiment of the present application.
Description of the drawings:
a distribution management apparatus 1; a data distribution device 2; a message queue entering module 4;
a dispenser 22; a distribution module 23; a message queue module 5;
a data service module 3; a data mapping unit 231; a data transmission unit 232;
a communication unit 25; a data distribution module 251; a first data receiving module 252;
a second data receiving module 221; a processor 81; a memory 82;
a communication interface 83; a bus 80.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described and illustrated below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present application. All other embodiments, which can be made by one of ordinary skill in the art without undue burden on the person of ordinary skill in the art based on the embodiments provided herein, are intended to be within the scope of the present application.
It is apparent that the drawings in the following description are only some examples or embodiments of the present application, and it is possible for those of ordinary skill in the art to apply the present application to other similar situations according to these drawings without inventive effort. Moreover, it should be appreciated that while such a development effort might be complex and lengthy, it would nevertheless be a routine undertaking of design, fabrication, or manufacture for those of ordinary skill having the benefit of this disclosure, and thus should not be construed as having the benefit of this disclosure.
Reference in the specification to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is to be expressly and implicitly understood by those of ordinary skill in the art that the embodiments described herein can be combined with other embodiments without conflict.
Unless defined otherwise, technical or scientific terms used herein should be given the ordinary meaning as understood by one of ordinary skill in the art to which this application belongs. Reference to "a," "an," "the," and similar terms herein do not denote a limitation of quantity, but rather denote the singular or plural. The terms "comprising," "including," "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, article, or apparatus that comprises a list of steps or modules (elements) is not limited to only those steps or elements but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus. The terms "connected," "coupled," and the like in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The term "plurality" as used herein refers to two or more. "and/or" describes an association relationship of an association object, meaning that there may be three relationships, e.g., "a and/or B" may mean: a exists alone, A and B exist together, and B exists alone. The character "/" generally indicates that the context-dependent object is an "or" relationship. The terms "first," "second," "third," and the like, as used herein, are merely distinguishing between similar objects and not representing a particular ordering of objects.
Message Queue (Message Queue), an important component in a distributed system, can be described simply as a general usage scenario:
when immediate results are not needed, but concurrency is controlled, almost the same time a message queue is used.
The message queue mainly solves the problems of application coupling, asynchronous processing, flow cutting and the like.
Currently more used message queues are RabbitMQ, rocketMQ, activeMQ, kafka, zeroMQ, metaMq, etc., and partial databases such as Redis, mysql, and phxsql may also implement the message queue functions.
In an actual application scene, the dimensions of real-time data services such as equipment state monitoring, early warning and the like for data analysis are different, and only one or a plurality of fields of the equipment data are actually used; whereas device data often has a large number of fields, the information of a single piece of data is large, and the frequency of data generation is high, and if the full amount of data is transmitted to each data service, a large amount of data transmission resources are consumed. There is therefore a need for a data distribution system to distribute device data, precisely sending the data dimensions it needs to each data service, to achieve high efficiency of data transmission.
On the other hand, the network environment where the industrial equipment is located is complex, the equipment data are easily influenced by the environment, the equipment data received at the acquisition end are generally unevenly distributed in time, a scene where a large number of data are concentrated arrives often occurs, and the processing performance and the I/O pressure are brought to the data transmission system. Therefore, the data distribution system must have high stability and fault tolerance.
Embodiment one:
the embodiment provides a data distribution method. Fig. 1 is a flowchart of a data distribution method according to an embodiment of the present application, as shown in fig. 1, the flowchart including the steps of:
a parameter configuration step S1, setting parameters of a distribution management device and starting a data distribution device, wherein the distribution management device generates a consumer of an incoming message queue, a producer of an outgoing message queue and a corresponding data mapping table according to the parameters;
the data flow in step S2, the distribution management device matches the streaming data source with a data model table and obtains each field of the streaming data source, and the streaming data source enters the data distribution device through an incoming message queue according to the consumer;
and step S3, the data distribution device selects corresponding fields in the streaming data source according to the data mapping table and sends the corresponding fields to the corresponding data service modules according to the producer.
Through the steps, the publish-subscribe distribution system is combined with the message queue, the data produced by the data source is distributed to the message queue dedicated to each data service through the publish-subscribe distribution system, and before the data is published to the message queue, the data mapping module is used for mapping the industrial equipment data, and only the required data dimension is written into the message queue, so that the resources consumed by redundant data in network transmission are reduced, and the data distribution efficiency is improved. On the other hand, the distribution management device distributes data to the message queues instead of directly to the consumers, and can disperse a part of pressure to the message queues when the data service consumers generate back pressure, thereby enhancing the stability of the system.
The method also effectively solves the problems of low redundancy efficiency and low system robustness of the existing publish-subscribe scheme, completes distribution and data mapping in the device, ensures that each consumer only obtains the required data dimension, improves the data transmission efficiency of the system, and simultaneously uses the message queue as a buffer for sending data to each consumer, and improves the stability of the system.
In some practical applications, before the data distribution device is started, a user may set parameters of the data distribution device through a front end page, where the settable parameter configuration includes an in-message queue, an out-message queue metadata, and a data mapping model, where:
the incoming message queue is a data channel connected with a streaming data source, and data enters the data distribution device from the incoming message queue;
the outgoing message queue metadata comprises the maximum duration of the data in the outgoing message queue, the number of fragments of the outgoing message queue and the number of copied fragments of the outgoing message queue;
the data mapping model is a method for converting a data model of a data source into a data model of an outgoing message queue by the data mapping module.
The data mapping table comprises a data model of a data source and a data model of an outgoing message queue, and the user distribution management device acquires corresponding fields of the data source through the data model.
In some of these embodiments, the parameters in the parameter configuration step S1 include a data model of the in-message queue, out-message queue metadata, and a mapping manner of each data mapping module.
In some embodiments, the data streaming step S2 specifically includes:
the streaming data source enters the distributor of the data distribution device through the fragments entering the message queue, and the distributor corresponds to the fragments one by one.
In practical application, the number of the slicers is set according to the number of the slicers in the message queue, and the slicers are in one-to-one correspondence to be used for transmitting data.
Fig. 2 is a flowchart of data distribution steps according to an embodiment of the present application, as shown in fig. 2, in some embodiments, the data distribution step S3 specifically includes the following steps:
a data receiving step S31, in which the streaming data sources are pushed to a corresponding number of distribution modules according to a preset distribution target number;
each distribution module corresponds to a task respectively, so that interference is avoided.
A field extraction step S32, wherein the data mapping module of each distribution module performs field extraction on the streaming data source according to the data mapping table;
and a data transmission step S33, wherein the extracted fields are transmitted to the data service module through the outgoing message queue according to the producer.
In practical applications, the distributor communicates with the distribution units, and the distributor pushes the streaming data source to each distribution module.
In some of these embodiments, the manner of connection between the dispenser and the dispensing module comprises:
in a distributed environment, the distributor and the distribution module communicate through a TCP protocol;
in a stand-alone environment, asynchronous communication between the dispenser and the dispensing module is performed through an observer mode;
the observer mode (sometimes also referred to as Model-View mode, source-Listener mode, or slave mode) is one of the software design modes.
In this mode, one target object manages all observer objects that depend on it and actively issues notifications when its own state changes. This is typically accomplished by calling the methods provided by each observer. Such a mode is commonly used to implement event handling systems.
In practical applications, in a distributed environment, the distributor and the distribution module are deployed in different processes and communicate using the TCP protocol; in a stand-alone environment, the dispatcher and the dispatcher module are located in different threads of the same process, and the observer mode is used for inter-thread asynchronous communication.
In some embodiments, the distributor and the distribution module record a state once every set time, and the distributor and the distribution module write the sent field into the same cache area while sending the field downstream;
in a distributed environment, after multiple retry failures, the distributor distributes and starts a new distribution unit by the distribution management device, and the distributor resends the data in the buffer to the new distribution unit.
The distributor and the distribution module record a state once every time a period of time passes, and the distributor and the distribution module write the sent data into a buffer area while sending the data to the downstream (the distribution module and the message outlet queue) for recovering from the last state in the case of failure. The buffer is emptied each time the status is recorded.
In the distributed environment, after the multiple retries fail, the distributor distributes and starts a new distribution module by the distribution management device, and the distributor resends the data in the buffer area to the new distribution module. Thereafter, even if the failed dispensing unit recovers after a period of time, the dispenser will not send data thereto.
The embodiments of the present application are described and illustrated below by means of preferred embodiments.
Fig. 3 is a flowchart of a data distribution method according to a preferred embodiment of the present application.
S301, a distribution management device parameter configuration step
Before the data distribution device is started, a user can set parameters of the data distribution device through a front page.
S302, generating configuration task step
The distribution management device is responsible for storing the data model of the incoming message queue (i.e. the data source), the metadata of the outgoing message queue, and the mapping mode of each data mapping module. After the user completes parameter setting and starts the data distribution device, the distribution management device generates a configuration task according to the parameters, wherein the configuration task comprises: the internal topology of the data distribution device, the consumer assigned to the message queue, and the producer of the outgoing message queue, and then initiates the device for data distribution.
S303, a distributing step
Under the distributed environment, data enter the distributor through the message-in queue, the data distribution module of the distributor pushes the data to the corresponding first data receiving module of each distribution unit, the data mapping module inside each distribution unit extracts the fields of the data, and the set data fields are sent to the data service module through the message-out queue.
The data distribution module and the first data receiving module are used here as communication functions for data transmission between the distributor and the distribution unit.
In a stand-alone environment, asynchronous communication between threads is realized between the distributor and the distribution unit through an observer mode, and other parts are the same and are not repeated.
S304, distributing fault-tolerant steps
Scene of failure of distribution module: in the distributed environment, after multiple retry failures, the distributor distributes and starts a new distribution module by the distribution management device, and the distributor resends the data in the buffer to the new distribution module. Even if the failed distribution module recovers after a period of time, the distributor no longer distributes data to it.
It should be noted that, according to the principle that data in the message queue can only be processed by one consumer, the failure of the distributor will not cause the loss of data, the data originally responsible for receiving by the distributor will be received and distributed by another distributor, and in this scenario, only the throughput of the system will be reduced, which is not described in detail in the present invention.
In addition, in a stand-alone environment, the distribution module and the distributor are located in the same process, and the independent faults are almost impossible, so the invention does not discuss the situation.
The steps shown in the flowcharts described above or in the flowcharts of the figures may be performed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowcharts, in some cases the steps shown or described may be performed in an order different from that herein.
The embodiment also provides a data distribution system, and the device is used for implementing the above embodiments and preferred embodiments, and will not be described in detail. As used below, the terms "module," "unit," "sub-unit," and the like may be a combination of software and/or hardware that implements a predetermined function. While the system described in the following embodiments is preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.
Fig. 4 is a block diagram of a data distribution system according to an embodiment of the present application, as shown in fig. 4, the system includes: the distribution management device 1, the data distribution device 2, the incoming message queue module 4 and the outgoing message queue module 5, wherein:
the distribution management apparatus 1 stores a data model table, a data map table, and outgoing message queue metadata;
the data distribution apparatus 2 includes a distributor 22 and a distribution module 23;
the distribution management device 1 matches the streaming data source according to the data model table and enters the streaming data source into the distributor 22 through the incoming message queue module 4;
the distribution module 23 extracts the streaming data source according to the data mapping table, and sends the extracted streaming data source to a data service module 3 through the outgoing message queue module 5 according to the outgoing message queue metadata;
in some of these embodiments, the dispenser 22 and the dispensing unit 23 are provided as one or more;
the distributor 22 corresponds to the fragments in the message queue module 4 one by one, and the streaming data sources in the fragments are correspondingly transmitted to the distributor 22;
the number of the distribution modules 23 is adapted to the preset distribution target number, the distribution modules 23 process the streaming data sources according to the corresponding distribution targets, and the processed streaming data sources are transmitted to the corresponding data service modules 3.
In practical applications, a data distribution device may comprise one or more distributors according to the number of fragments in the incoming message queue; depending on the number of distribution targets, one or more distribution units may be included. Each distributor will send the received data to all distribution units.
The data distribution mode adopts a pushing mode, and a data distribution module in the distributor pushes a data stream to a data receiving module of each distribution unit.
In some of these embodiments, the distribution module 23 comprises a data mapping unit 231 and a data sending unit 232, wherein:
the data mapping unit 231 converts the data model of the streaming data source into the data model of the outgoing message queue according to the data mapping table, and transmits the processed streaming data source to the corresponding data transmitting unit;
the data transmitting unit 232 transmits the streaming data source to the corresponding outgoing message queue module 5.
In a distributed environment, the distributor 22 and the distribution module 23 are located in different processes; in a stand-alone environment, the dispatcher 22 and the dispatcher module 23 are located in different threads of the same process.
Fig. 5 is a block diagram of a preferred structure of a data distribution system according to an embodiment of the present application, and as shown in fig. 5, the system includes all the modules shown in fig. 4, and further includes a communication unit 25, specifically:
in a distributed environment, the communication unit 25 is disposed on the dispenser 22 and the dispensing module 23, and includes a data dispensing module 251 and a first data receiving module 252, which are respectively located on the dispenser 22 and the dispensing unit 23. The distributor 22 and the distribution module 23 communicate via the TCP protocol.
The distributor 22 further comprises a second data receiving module 221 for receiving data from the incoming message queue,
in a stand-alone environment, the dispatcher 22 and the dispatcher module 23 communicate asynchronously using the observer mode without adding an additional module for communication.
The above-described respective modules may be functional modules or program modules, and may be implemented by software or hardware. For modules implemented in hardware, the various modules described above may be located in the same processor; or the above modules may be located in different processors in any combination.
In addition, the data distribution method of the embodiment of the present application described in connection with fig. 1 may be implemented by a computer device. Fig. 6 is a schematic diagram of a hardware structure of a computer device according to an embodiment of the present application.
The computer device may include a processor 81 and a memory 82 storing computer program instructions.
In particular, the processor 81 may include a Central Processing Unit (CPU), or an application specific integrated circuit (Application Specific Integrated Circuit, abbreviated as ASIC), or may be configured to implement one or more integrated circuits of embodiments of the present application.
Memory 82 may include, among other things, mass storage for data or instructions. By way of example, and not limitation, memory 82 may comprise a Hard Disk Drive (HDD), floppy Disk Drive, solid state Drive (Solid State Drive, SSD), flash memory, optical Disk, magneto-optical Disk, tape, or universal serial bus (Universal Serial Bus, USB) Drive, or a combination of two or more of the foregoing. The memory 82 may include removable or non-removable (or fixed) media, where appropriate. The memory 82 may be internal or external to the data processing apparatus, where appropriate. In a particular embodiment, the memory 82 is a Non-Volatile (Non-Volatile) memory. In a particular embodiment, the Memory 82 includes Read-Only Memory (ROM) and random access Memory (Random Access Memory, RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (Programmable Read-Only Memory, abbreviated PROM), an erasable PROM (Erasable Programmable Read-Only Memory, abbreviated EPROM), an electrically erasable PROM (Electrically Erasable Programmable Read-Only Memory, abbreviated EEPROM), an electrically rewritable ROM (Electrically Alterable Read-Only Memory, abbreviated EAROM), or a FLASH Memory (FLASH), or a combination of two or more of these. The RAM may be Static Random-Access Memory (SRAM) or dynamic Random-Access Memory (Dynamic Random Access Memory DRAM), where the DRAM may be a fast page mode dynamic Random-Access Memory (Fast Page Mode Dynamic Random Access Memory FPMDRAM), extended data output dynamic Random-Access Memory (Extended Date Out Dynamic Random Access Memory EDODRAM), synchronous dynamic Random-Access Memory (Synchronous Dynamic Random-Access Memory SDRAM), or the like, as appropriate.
Memory 82 may be used to store or cache various data files that need to be processed and/or communicated, as well as possible computer program instructions for execution by processor 81.
The processor 81 implements any of the data distribution methods of the above embodiments by reading and executing computer program instructions stored in the memory 82.
In some of these embodiments, the computer device may also include a communication interface 83 and a bus 80. As shown in fig. 6, the processor 81, the memory 82, and the communication interface 83 are connected to each other through the bus 80 and perform communication with each other.
The communication interface 83 is used to implement communications between various modules, devices, units, and/or units in embodiments of the present application. Communication port 83 may also enable communication with other components such as: and the external equipment, the image/data acquisition equipment, the database, the external storage, the image/data processing workstation and the like are used for data communication.
Bus 80 includes hardware, software, or both, coupling components of the computer device to each other. Bus 80 includes, but is not limited to, at least one of: data Bus (Data Bus), address Bus (Address Bus), control Bus (Control Bus), expansion Bus (Expansion Bus), local Bus (Local Bus). By way of example, and not limitation, bus 80 may include a graphics acceleration interface (Accelerated Graphics Port), abbreviated AGP, or other graphics Bus, an enhanced industry standard architecture (Extended Industry Standard Architecture, abbreviated EISA) Bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an industry standard architecture (Industry Standard Architecture, ISA) Bus, a wireless bandwidth (InfiniBand) interconnect, a Low Pin Count (LPC) Bus, a memory Bus, a micro channel architecture (Micro Channel Architecture, abbreviated MCa) Bus, a peripheral component interconnect (Peripheral Component Interconnect, abbreviated PCI) Bus, a PCI-Express (PCI-X) Bus, a serial advanced technology attachment (Serial Advanced Technology Attachment, abbreviated SATA) Bus, a video electronics standards association local (Video Electronics Standards Association Local Bus, abbreviated VLB) Bus, or other suitable Bus, or a combination of two or more of the foregoing. Bus 80 may include one or more buses, where appropriate. Although embodiments of the present application describe and illustrate a particular bus, the present application contemplates any suitable bus or interconnect.
The computer device may perform the parameter configuration step in the embodiment of the present application based on the acquired parameter of the distribution management apparatus, thereby implementing the data distribution method described in connection with fig. 1.
In addition, in combination with the data distribution method in the above embodiment, the embodiment of the application may be implemented by providing a computer readable storage medium. The computer readable storage medium has stored thereon computer program instructions; the computer program instructions, when executed by a processor, implement any of the data distribution methods of the above embodiments.
Embodiment two:
on the basis of the technical scheme of the first embodiment, the outgoing message queue can be replaced by a database system.
The reason for this is that in the first embodiment, the data in the outgoing message queue is generated by the data mapping module and has a fixed data structure, so that the outgoing message queue can be logically regarded as a database with the function of a message queue. When data needs to be distributed to a persistent storage system, the outgoing message queue may be replaced directly with the database system.
The technical features of the above-described embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above-described embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The above examples merely represent a few embodiments of the present application, which are described in more detail and are not to be construed as limiting the scope of the invention. It should be noted that it would be apparent to those skilled in the art that various modifications and improvements could be made without departing from the spirit of the present application, which would be within the scope of the present application. Accordingly, the scope of protection of the present application is to be determined by the claims appended hereto.
Claims (9)
1. A data distribution method, applied to a publish-subscribe data distribution system, comprising the steps of:
setting parameters of a distribution management device and starting the data distribution device, wherein the distribution management device generates a consumer of an incoming message queue, a producer of an outgoing message queue and a corresponding data mapping table according to the parameters, the data mapping table comprises a data model of a streaming data source and a data model of an outgoing message queue, the parameters of the distribution management device comprise the data model of the incoming message queue, metadata of the outgoing message queue and a mapping mode of each data mapping module, a method for converting the data model of the streaming data source into the data model of the outgoing message queue is a data mapping model, the data mapping model is set before the data distribution device is started, and the incoming message queue is a data channel of an incoming streaming data source;
a data inflow step, wherein the distribution management device matches a streaming data source with a data model table and obtains each field of the streaming data source, and the streaming data source enters the data distribution device through the incoming message queue according to the consumer;
and a data distribution step, wherein the data distribution device selects the corresponding field in the stream data source according to the data model of the outgoing message queue of the data mapping table, and sends the corresponding field to a corresponding data service module according to the producer.
2. The data distribution method according to claim 1, wherein the data inflow step specifically includes:
the streaming data source enters a distributor of the data distribution device through the fragments of the incoming message queue, and the distributor corresponds to the fragments one by one.
3. The data distribution method according to claim 2, characterized in that the data distribution step specifically comprises the steps of:
a data receiving step, namely pushing the streaming data source to a corresponding number of distribution modules according to a preset distribution target number;
a field extraction step, wherein a data mapping module of each distribution module performs field extraction on the streaming data source according to the data mapping table;
and a data sending step, wherein the extracted fields are sent to the data service module through the message output queue according to the producer.
4. A data distribution method according to claim 3, wherein the manner of connection between the distributor and the distribution module comprises:
in a distributed environment, the distributor and the distribution module communicate through a TCP protocol;
in a stand-alone environment, asynchronous communication between the dispatcher and the dispatch module is performed through an observer mode.
5. The method for distributing data according to any one of claims 1 to 4, wherein,
the distributor and the distribution module record a state once every time a set time passes, and the distributor and the distribution module write the sent field into the same cache area while sending the field downstream;
in a distributed environment, after multiple retry failures, the distributor distributes and starts a new distributing unit by the distributing management device, and the distributor resends the data in the buffer to the new distributing unit.
6. A data distribution system, applying the data distribution method according to any one of claims 1-5, characterized in that it comprises a distribution management device, a data distribution device, an in-message queue module and an out-message queue module, wherein:
the distribution management device stores a data model table, a data mapping table and outgoing message queue metadata;
the data distribution device comprises a distributor and a distribution module;
the distribution management device is matched with a streaming data source according to the data model table, and the streaming data source enters the distributor through the message-entering queue module;
the distribution module extracts the streaming data source according to the data mapping table, and sends the extracted streaming data source to a data service module through the message-out queue module according to the message-out queue metadata.
7. The data distribution system of claim 6, wherein the distributor and distribution unit are provided in one or more of the following;
the distributor corresponds to the fragments in the message queue module one by one, and the streaming data sources in the fragments are correspondingly transmitted to the distributor;
the number of the distribution modules is adapted to the preset distribution target number, the distribution modules process the streaming data sources according to the corresponding distribution targets, and the processed streaming data sources are transmitted to the corresponding data service modules.
8. The data distribution system of claim 7, wherein the distribution module comprises a data mapping unit and a data transmitting unit, wherein:
the data mapping unit converts the data model of the streaming data source into a data model of an outgoing message queue according to the data mapping table, and sends the processed streaming data source to the corresponding data sending unit;
and the data sending unit sends the streaming data source to the corresponding message outlet queue module.
9. A data distribution system according to any of claims 6-8, comprising:
in a distributed environment, a communication unit is arranged between the distributor and the distribution module, and the distributor and the distribution module are communicated through a TCP protocol;
in a stand-alone environment, the observer mode is used for asynchronous communication.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110105058.2A CN112953757B (en) | 2021-01-26 | 2021-01-26 | Data distribution method, system and computer equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110105058.2A CN112953757B (en) | 2021-01-26 | 2021-01-26 | Data distribution method, system and computer equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112953757A CN112953757A (en) | 2021-06-11 |
| CN112953757B true CN112953757B (en) | 2023-12-29 |
Family
ID=76237095
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110105058.2A Active CN112953757B (en) | 2021-01-26 | 2021-01-26 | Data distribution method, system and computer equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112953757B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113342867A (en) * | 2021-07-05 | 2021-09-03 | 招商局金融科技有限公司 | Data distribution and management method and device, electronic equipment and readable storage medium |
| CN115567557B (en) * | 2022-09-20 | 2024-04-16 | 中国联合网络通信集团有限公司 | Multi-target reporting method, device and system for industrial data |
| CN116894649B (en) * | 2023-09-11 | 2023-11-17 | 成都秦川物联网科技股份有限公司 | Industrial Internet of things main service platform data distribution system, method and medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103973809A (en) * | 2014-05-22 | 2014-08-06 | 北京锐安科技有限公司 | Data distributing method and system |
| CN105959151A (en) * | 2016-06-22 | 2016-09-21 | 中国工商银行股份有限公司 | High availability stream processing system and method |
| CN106339435A (en) * | 2016-08-19 | 2017-01-18 | 中国银行股份有限公司 | Data distribution method, device and system |
| CN111866082A (en) * | 2020-06-22 | 2020-10-30 | 远光软件股份有限公司 | Data distribution method and device based on target system configuration |
| CN111913821A (en) * | 2020-08-17 | 2020-11-10 | 武汉众邦银行股份有限公司 | Method for realizing cross-data-source real-time data stream production consumption |
-
2021
- 2021-01-26 CN CN202110105058.2A patent/CN112953757B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103973809A (en) * | 2014-05-22 | 2014-08-06 | 北京锐安科技有限公司 | Data distributing method and system |
| CN105959151A (en) * | 2016-06-22 | 2016-09-21 | 中国工商银行股份有限公司 | High availability stream processing system and method |
| CN106339435A (en) * | 2016-08-19 | 2017-01-18 | 中国银行股份有限公司 | Data distribution method, device and system |
| CN111866082A (en) * | 2020-06-22 | 2020-10-30 | 远光软件股份有限公司 | Data distribution method and device based on target system configuration |
| CN111913821A (en) * | 2020-08-17 | 2020-11-10 | 武汉众邦银行股份有限公司 | Method for realizing cross-data-source real-time data stream production consumption |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112953757A (en) | 2021-06-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112953757B (en) | Data distribution method, system and computer equipment | |
| CN109739929B (en) | Data synchronization method, device and system | |
| CN109542865B (en) | Method, device, system and medium for synchronizing configuration files of distributed cluster system | |
| CN108712332B (en) | Communication method, system and device | |
| CN109376197B (en) | Data synchronization method, server and computer storage medium | |
| CN110601903B (en) | Data processing method and device based on message queue middleware | |
| CN111831748B (en) | Data synchronization method, device and storage medium | |
| CN114363407B (en) | Message service method and device, readable storage medium and electronic equipment | |
| CN105760519A (en) | Cluster file system and file lock allocation method thereof | |
| CN111399764B (en) | Data storage method, data reading device, data storage equipment and data storage medium | |
| CN113742031A (en) | Node state information acquisition method and device, electronic equipment and readable storage medium | |
| CN109167819B (en) | Data synchronization system, method, device and storage medium | |
| JP2021524104A (en) | Master / Standby Container System Switching | |
| EP3713195B1 (en) | Log processing method, related device, and system | |
| CN113703954A (en) | Message backup method and device, electronic equipment and computer storage medium | |
| CN111541762B (en) | Data processing method, management server, device and storage medium | |
| CN110138753B (en) | Distributed message service system, method, apparatus, and computer-readable storage medium | |
| CN112804332A (en) | Message processing system, method, device, equipment and computer readable storage medium | |
| CN114765706A (en) | Method and device for triggering vOMCI function from OLT to send OMCI message | |
| CN109614354B (en) | Interface distribution method, board card, equipment and storage medium | |
| CN111427689A (en) | Cluster keep-alive method and device and storage medium | |
| CN110807058B (en) | Method and system for exporting data | |
| CN115629909A (en) | Service data processing method and device, electronic equipment and storage medium | |
| CN114661526A (en) | Data backup method and device | |
| CN114546926B (en) | Core cluster synchronization, control method, data processing method, core, device and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |