CN111917814A - Data publishing method, data subscribing method, data publishing device, data subscribing system and readable storage medium - Google Patents
Data publishing method, data subscribing method, data publishing device, data subscribing system and readable storage medium Download PDFInfo
- Publication number
- CN111917814A CN111917814A CN201910390171.2A CN201910390171A CN111917814A CN 111917814 A CN111917814 A CN 111917814A CN 201910390171 A CN201910390171 A CN 201910390171A CN 111917814 A CN111917814 A CN 111917814A
- Authority
- CN
- China
- Prior art keywords
- data
- publishing
- subscription
- memory
- storage information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
Abstract
The embodiment of the invention discloses a method, a device, equipment and a system for publishing and subscribing data and a readable storage medium. The data publishing method is applied to a publishing agent configured in publishing equipment and comprises the following steps: acquiring data to be issued generated by the issuing equipment; storing the data in a memory; and sending the storage information of the data in the memory to the coordination service equipment so that a subscription agent configured in the subscription equipment acquires the storage information from the coordination service equipment, and subscribing the corresponding data from the memory according to the storage information. The embodiment of the invention reduces the consumption of machine resources and reduces the data transmission delay.
Description
Technical Field
Embodiments of the present invention relate to data transmission technologies, and in particular, to a method, an apparatus, a device, a system, and a readable storage medium for publishing and subscribing data.
Background
At present, a large amount of data is generated by more and more working machines, and the value of the data is more and more emphasized. Two main aspects of calculation are currently performed on the data in the field of big data: real-time streaming calculation and offline batch calculation. In either case, the data needs to be transferred from the work machine to a downstream computer for further processing.
In the existing streaming system, data is published to message middleware, such as Kafka, Bigpipe, etc., from an operating machine, and a downstream computer subscribes to the data from the message middleware. However, the configuration of message middleware requires a large amount of machine resources to be consumed; also, message middleware causes data transmission delay.
Disclosure of Invention
Embodiments of the present invention provide a data publishing method, a data subscribing method, a data publishing device, a data subscribing device, a data publishing system, a data subscribing system, and a readable storage medium, so as to reduce consumption of machine resources and reduce data transmission delay.
In a first aspect, an embodiment of the present invention provides a data publishing method, which is applied to a publishing device
A publishing agent, comprising:
acquiring data to be issued generated by the issuing equipment;
storing the data in a memory;
and sending the storage information of the data in the memory to the coordination service equipment so that a subscription agent configured in the subscription equipment acquires the storage information from the coordination service equipment, and subscribing the corresponding data from the memory according to the storage information.
In a second aspect, an embodiment of the present invention further provides a data subscription method, applied to a subscription broker configured in a subscription device, including:
acquiring storage information of data to be subscribed in a memory from coordination service equipment;
according to the storage information, corresponding data are subscribed from a memory;
the data to be subscribed is acquired by a publishing agent configured in publishing equipment and is stored in the memory; and the storage information is sent to the coordination service equipment by the publishing agent.
In a third aspect, an embodiment of the present invention further provides a data publishing apparatus, which is applied to a publishing agent configured in a publishing device, where the apparatus includes:
the acquisition module is used for acquiring the data to be issued generated by the issuing equipment;
the storage module is used for storing the data into a memory;
and the sending module is used for sending the storage information of the data in the memory to the coordination service equipment so that a subscription agent configured in the subscription equipment can acquire the storage information from the coordination service equipment, and subscribe the corresponding data from the memory according to the storage information.
In a fourth aspect, an embodiment of the present invention further provides a data subscription apparatus, which is applied to a subscription broker configured in a subscription device, where the apparatus includes:
the acquisition module is used for acquiring the storage information of the data to be subscribed in the memory from the coordination service equipment;
the subscription module is used for subscribing corresponding data from the memory according to the storage information;
the data to be subscribed is acquired by a publishing agent configured in publishing equipment and is stored in the memory; and the storage information is sent to the coordination service equipment by the publishing agent.
In a fifth aspect, an embodiment of the present invention further provides a publishing device, including
One or more processors;
a memory for storing one or more programs,
when the one or more programs are executed by the one or more processors, the one or more processors implement the data distribution method of any of the embodiments.
In a sixth aspect, an embodiment of the present invention further provides a subscription device, including
One or more processors;
a memory for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the data subscription method of any embodiment.
In a seventh aspect, an embodiment of the present invention further provides a data publishing and subscribing system, including: coordinating a service device, a publishing device in any embodiment and a subscribing device in any embodiment;
the coordination service equipment is used for receiving and storing the storage information of the data sent by the issuing equipment in the memory; and sending the storage information to the subscribing device.
In an eighth aspect, an embodiment of the present invention further provides a storage medium readable by a subscribing device, where the storage medium stores a subscribing device program, and the program, when executed by a processor, implements the data publishing method according to any embodiment or the data subscribing method according to any embodiment.
In the embodiment of the invention, the publishing device is provided with the publishing agent, the subscribing device is provided with the subscribing agent, and devices such as additional message middleware and the like do not need to be added outside the subscribing device and the publishing device, so that the resource consumption of a machine is reduced; the data to be published is stored in the memory through the publishing agent, the stored information is sent to the coordination service device, and the subscription agent subscribes from the memory according to the stored information in the coordination service device, so that the coordination service device is used as an intermediate medium of the publishing agent and the subscription agent, information interaction is performed between the publishing agent and the subscription agent, the publishing and the subscription of the data are completed, transfer of third-party devices such as message intermediate pieces is not needed, data transmission delay is reduced, and data transmission efficiency is improved.
Drawings
Fig. 1 is a flowchart of a data publishing method according to an embodiment of the present invention;
fig. 2a is a flowchart of a data publishing method according to a second embodiment of the present invention;
fig. 2b is a schematic diagram of a data publishing process according to a second embodiment of the present invention;
fig. 3 is a flowchart of a data subscription method according to a third embodiment of the present invention;
fig. 4a is a flowchart of a data subscription method according to a fourth embodiment of the present invention;
fig. 4b is a schematic diagram of a data subscription process according to a fourth embodiment of the present invention;
fig. 5 is a schematic structural diagram of a data distribution apparatus according to a fifth embodiment of the present invention;
fig. 6 is a schematic structural diagram of a data subscription device according to a sixth embodiment of the present invention;
fig. 7 is a schematic structural diagram of a publishing device according to a seventh embodiment of the present invention;
fig. 8 is a schematic structural diagram of an eighth data publishing and subscribing system according to an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Example one
Fig. 1 is a flowchart of a data publishing method according to an embodiment of the present invention, where the embodiment is applicable to a scenario of streaming data. The embodiment is applied to the publishing agent configured in the publishing device. The publishing device is any device capable of generating data, such as a mobile terminal, a PC, a backend server, etc. In this embodiment, a publishing agent (broker), such as mosquitto and emqttd, is configured inside the publishing device, and the publishing agent completes the data publishing operation. The method can be executed by a data distribution device, which can be formed by hardware and/or software and is generally integrated in a distribution device, and specifically includes the following operations:
s110, data to be published generated by the publishing device are obtained.
The data generated by the publishing device includes, but is not limited to, logs, business data, heartbeat data, and the like. The publishing agent may obtain the data to be published in real time or periodically.
And S120, storing the data into a memory.
The publishing agent stores the retrieved data in memory. Optionally, the memory includes a cache and/or a distributed file system. The cache is configured in the publishing agent; the Distributed File System (DFS) is configured on the publishing device or independent from the publishing device, i.e. configured on a device other than the publishing device and the subscribing device.
Based on the difference between the storage modes of the distributed file system and the cache, the publishing agent is configured to store the data with low real-time performance into the distributed file system and back up the data, so that offline batch subscription can be conveniently carried out on the data. The data with high real-time performance is stored in the cache, and efficient and quick data subscription can be realized. Of course, the memory may also include various forms of media such as solid-state memory, flash memory, etc., and the embodiment is not limited to cache and distributed file systems.
S130, sending the storage information of the data in the memory to the coordination service equipment so that the subscription agent configured in the subscription equipment can acquire the storage information from the coordination service equipment, and subscribing the corresponding data from the memory according to the storage information.
The storage information comprises data storage progress and storage position. The data storage progress comprises whether the data is stored or not and the percentage of the data being stored; the storage location includes data storage in a cache or in a distributed file system, and a specific storage path.
And the release agent sends the stored information to the coordination service equipment in real time in the process of storing the data into the memory so as to ensure that the stored information in the coordination service equipment is updated in time.
The coordination service device is configured with software, such as a zookeeper, that provides a consistency service for the publish broker and the subscribe broker. The coordination service device is in communication connection with the publish broker and the subscribe broker respectively, for example, by heartbeat mutual information, but not limited to other communication modes. The coordination service equipment is used for receiving and storing the storage information of the data sent by the publishing agent in the memory; and sending the stored information to the subscription broker.
The subscription agent is configured in the subscription equipment, acquires the storage information from the coordination service equipment, and subscribes corresponding data from the memory according to the storage information. Then, the subscribed data is computed in the subscribing device.
In the embodiment of the invention, the publishing device is provided with the publishing agent, the subscribing device is provided with the subscribing agent, and devices such as additional message middleware and the like do not need to be added outside the subscribing device and the publishing device, so that the resource consumption of a machine is reduced; the data to be published is stored in the memory through the publishing agent, the stored information is sent to the coordination service device, and the subscription agent subscribes from the memory according to the stored information in the coordination service device, so that the coordination service device is used as an intermediate medium of the publishing agent and the subscription agent, information interaction is performed between the publishing agent and the subscription agent, the publishing and the subscription of the data are completed, transfer of third-party devices such as message intermediate pieces is not needed, data transmission delay is reduced, and data transmission efficiency is improved.
Furthermore, due to the above information interaction mode of the publish broker and the subscribe broker, the message middleware is omitted, so that the subscribe broker can autonomously select a real-time subscription mode and an offline batch subscription mode, and is not limited by the subscription mode of the message middleware, and the data subscription is applicable to at least one downstream device and different subscription modes.
In the foregoing embodiment and the following embodiments, the acquiring data to be published generated by the publishing device includes: and receiving a publishing task sent by the management equipment, and acquiring the data to be published generated by the publishing equipment according to the publishing task.
Wherein the management device (master) is communicatively coupled to the publishing agent and to the system database. The system database stores the release tasks set by users or release devices, and the management device sends the release tasks stored in the system database to the release agent. The publishing task includes data information to be published, such as data of a certain period of time, data of a certain service, data of a certain subject, and the like, so that the publishing agent obtains the data to be published generated by the publishing device according to the publishing task.
Further optionally, after the storage information of the data in the memory is sent to the coordination service device, the data publishing operation is completed, and a data publishing progress is reported to the management device, for example, which data is published, and the published data accounts for the percentage of the total published data. And the management equipment stores the data release progress into a system database. Specifically, after the management device slices the data release progress, the fragments are written into the system database, so that the data release progress is stored in the fragments, and the storage efficiency is improved.
In the foregoing embodiment and the following embodiments, during the process of executing the data publishing method by the publishing agent, the method further includes reporting the process flow progress and the process status to the monitoring device. Wherein, the processing flow progress comprises: and acquiring the process progress, the process progress of storing the process progress and the process progress of sending the stored information of the data to be issued. The processing states include: the unprocessed state, the processed state, and the processed state of each of the aforementioned process flows.
In the embodiment, the release task is configured by setting the management equipment, and the release progress is obtained; and the monitoring device is arranged to monitor the progress and the processing state of the processing flow, so that the accuracy and the reliability of the data issuing process are improved, and the maintenance of the issuing process is facilitated.
Example two
This embodiment is further optimized for the above-described embodiment. In this embodiment, an Application Programming Interface (API) is configured for the publishing agent on a logical level, and the data publishing method is implemented by Programming the publishing Interface. Fig. 2a is a flowchart of a data publishing method according to a second embodiment of the present invention, and fig. 2b is a schematic diagram of a data publishing process according to the second embodiment of the present invention. With reference to fig. 2a and fig. 2b, the method provided in this embodiment specifically includes the following operations:
s210, the release agent receives the release task sent by the management device through the release interface, and acquires the data to be released, which is generated by the release device, according to the release task.
Wherein, the issuing interface is connected with the management device in a communication way. And the management equipment sends the release tasks stored in the system database to the release interface. And after receiving the publishing task, the publishing interface acquires the corresponding data to be published from the publishing equipment.
S220, the publishing agent reads in the data to be published through the publishing interface and stores the read data in a memory. The memory includes a distributed file system and a cache.
And S230, the publishing agent sends the storage information to the coordination service equipment through the publishing interface.
Specifically, after the publishing agent stores the read data into the memory, the storage information is updated, the storage information is transmitted to the publishing interface, and then the publishing interface sends the storage information to the coordination service device.
Optionally, after S230, the publishing agent reports the data publishing progress to the management device through the publishing interface, and after the management device slices the data publishing progress, the slices are written into the system database.
Optionally, in the process of executing the data publishing method by the publishing agent, the process flow progress and the process state are reported to the monitoring device through the publishing interface.
In the embodiment, the data publishing method can be realized by programming the publishing interface and matching the publishing agent, the internal logic of the publishing agent does not need to be modified, and the publishing process is convenient to maintain and monitor.
EXAMPLE III
Fig. 3 is a flowchart of a data subscription method according to a third embodiment of the present invention, where the present embodiment is applicable to a scenario of streaming data. The embodiment is applied to the subscription broker configured in the subscribing device. A subscribing device is any device capable of computing data. In this embodiment, a subscription broker (broker), such as mosquitto and emqttd, is configured inside the subscription device, and the subscription broker completes the data subscription operation. The method may be performed by a data subscribing device, which may be formed by hardware and/or software and is generally integrated in a subscribing device, and specifically includes the following operations:
and S310, acquiring storage information of the data to be subscribed in the memory from the coordination service equipment.
The data to be subscribed are acquired by a publishing agent configured in the publishing equipment and stored in a memory; the storage information is sent to the coordination service device by the publishing agent.
And S320, according to the storage information, subscribing corresponding data from the memory.
Specifically, the subscription broker acquires a data storage progress from the coordination service device, determines the stored data according to the data storage progress, and further acquires a storage location of the stored data. Then, corresponding data is subscribed from the memory according to the storage position.
In the embodiment of the invention, the publishing device is provided with the publishing agent, the subscribing device is provided with the subscribing agent, and devices such as additional message middleware and the like do not need to be added outside the subscribing device and the publishing device, so that the resource consumption of a machine is reduced; the data to be published is stored in the memory through the publishing agent, the stored information is sent to the coordination service device, and the subscription agent subscribes from the memory according to the stored information in the coordination service device, so that the coordination service device is used as an intermediate medium of the publishing agent and the subscription agent, information interaction is performed between the publishing agent and the subscription agent, the publishing and the subscription of the data are completed, transfer of third-party devices such as message intermediate pieces is not needed, data transmission delay is reduced, and data transmission efficiency is improved.
Furthermore, due to the above information interaction mode of the publish broker and the subscribe broker, the message middleware is omitted, so that the subscribe broker can autonomously select a real-time subscription mode and an offline batch subscription mode, and is not limited by the subscription mode of the message middleware, and the data subscription is applicable to at least one downstream device and different subscription modes.
In the foregoing embodiment and the following embodiments, acquiring, from the coordination service device, storage information of data to be subscribed in the memory includes: receiving a subscription task sent by a management device, and determining data information to be subscribed and a subscription mode according to the subscription task, wherein the subscription mode comprises a real-time subscription mode or an offline batch subscription mode; and determining the storage information of the data to be subscribed in the memory according to the data information to be subscribed.
The management device sends the subscription tasks stored in the system database to the subscription agent. The subscription task comprises data information to be subscribed, such as data of a certain period, data of a certain service, data of a certain theme and the like; the subscription task also comprises a real-time subscription mode or an offline batch subscription mode. Then, the subscription agent acquires the storage information of the data to be subscribed in the memory from the coordination service equipment according to the data information to be subscribed.
Further optionally, subscribing to the corresponding data from the memory according to the storage information includes: and according to the storage information, subscribing corresponding data from the memory according to a subscription mode.
Further optionally, after subscribing the corresponding data from the memory according to the storage information, the data subscription operation is completed, and a data subscription progress, for example, which data is subscribed, is reported to the management device, where the subscribed data accounts for a percentage of the total amount of the subscribed data. And the management equipment stores the data subscription progress into a system database. Specifically, after the management device slices the data subscription progress, the fragments are written into the system database, so that the data subscription progress fragments are stored, and the storage efficiency is improved.
In the foregoing embodiment and the following embodiments, during the process of executing the data subscription method by the subscription broker, the method further includes reporting a processing flow progress and a processing state to the monitoring device. Wherein, the processing flow progress comprises: and storing the information acquisition process progress and the information subscription process progress. The processing states include: the unprocessed state, the processed state, and the processed state of each of the aforementioned process flows.
In the embodiment, a subscription task is configured by setting the management equipment, and a subscription progress is obtained; and the monitoring device is arranged to monitor the progress and the processing state of the processing flow, so that the accuracy and the reliability of the data subscription process are improved, and the maintenance of the subscription process is facilitated.
Example four
This embodiment is further optimized for the above-described embodiment. In this embodiment, a subscription Interface (API) is configured for the subscription broker in a logic level, and the data subscription method is implemented by Programming the subscription Interface. Fig. 4a is a flowchart of a data subscription method according to a fourth embodiment of the present invention, and fig. 4b is a schematic diagram of a data subscription process according to the fourth embodiment of the present invention. With reference to fig. 4a and 4b, the method provided in this embodiment specifically includes the following operations:
s410, the subscription agent acquires the storage information of the data to be subscribed in the memory from the coordination service equipment through the subscription interface.
The subscription interface is in communication connection with the management device. And the management equipment sends the subscription tasks stored in the system database to the subscription interface. And after receiving the subscription task, the subscription interface determines the data information to be subscribed and the subscription mode, and acquires the storage information from the coordination service equipment according to the data information to be subscribed.
Specifically, the subscription interface first obtains the access right of the coordination service device, and then obtains the storage information from the coordination service device.
And S420, the subscription agent subscribes corresponding data from the memory according to the storage information and transmits the data to the subscription interface.
The subscription interface transmits the storage information and the subscription mode to the subscription agent, and the subscription agent subscribes corresponding data from the memory according to the storage information and the subscription mode. The subscription interface reads the subscribed data from the subscription agent device, so that the subscription device pulls the subscribed data through the subscription interface and calculates.
Optionally, after S420, the subscription agent reports the data subscription progress to the management device through the subscription interface, and after the management device slices the data subscription progress, the slices are written into the system database.
Optionally, after S420, the subscription broker adds a process through the subscription interface, so as to continue to acquire the storage information of the next piece of data from the coordination service device through the added process, and subscribe the corresponding data from the memory according to the storage information of the next piece of data. Further optionally, the added process is persisted to effectively avoid the problem of data loss caused by process exit, and data recovery can be realized by utilizing the process persisted before when the next restart is carried out.
Optionally, in the process of executing the data subscription method by the subscription broker, the process flow progress and the process state are reported to the monitoring device through the subscription interface.
In the embodiment, the data subscription method can be realized by programming the subscription interface and matching with the subscription agent, the internal logic of the subscription agent does not need to be modified, and the maintenance and monitoring of the subscription process are facilitated.
EXAMPLE five
Fig. 5 is a schematic structural diagram of a data distribution apparatus according to a fifth embodiment of the present invention. The embodiment of the invention is suitable for a scene of streaming data, and is applied to the publishing agent configured in the publishing equipment. With reference to fig. 5, the apparatus provided in this embodiment includes: an acquisition module 51, a storage module 52 and a transmission module 53.
An obtaining module 51, configured to obtain data to be published, which is generated by a publishing device;
a storage module 52 for storing data into a memory;
and a sending module 53, configured to send storage information of the data in the memory to the coordination service device, so that a subscription broker configured in the subscription device obtains the storage information from the coordination service device, and subscribes corresponding data from the memory according to the storage information.
In the embodiment of the invention, the publishing device is provided with the publishing agent, the subscribing device is provided with the subscribing agent, and devices such as additional message middleware and the like do not need to be added outside the subscribing device and the publishing device, so that the resource consumption of a machine is reduced; the data to be published is stored in the memory through the publishing agent, the stored information is sent to the coordination service device, and the subscription agent subscribes from the memory according to the stored information in the coordination service device, so that the coordination service device is used as an intermediate medium of the publishing agent and the subscription agent, information interaction is performed between the publishing agent and the subscription agent, the publishing and the subscription of the data are completed, transfer of third-party devices such as message intermediate pieces is not needed, data transmission delay is reduced, and data transmission efficiency is improved.
Optionally, when acquiring the data to be published generated by the publishing device, the acquiring module 51 is specifically configured to: and receiving a publishing task sent by the management equipment, and acquiring the data to be published generated by the publishing equipment according to the publishing task.
Optionally, the apparatus further includes a reporting module, configured to report a data publishing progress to the management device after sending storage information of the data in the storage to the coordination service device.
The data issuing device provided by the embodiment of the invention can execute the data issuing method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
EXAMPLE six
Fig. 6 is a schematic structural diagram of a data subscription device according to a sixth embodiment of the present invention. The embodiment of the invention is suitable for a scene of streaming data, and is applied to a subscription agent configured in subscription equipment. With reference to fig. 6, the apparatus provided in this embodiment includes: an acquisition module 61 and a subscription module 62.
An obtaining module 61, configured to obtain, from the coordination service device, storage information of data to be subscribed in a memory;
a subscription module 62, configured to subscribe corresponding data from the memory according to the storage information;
the data to be subscribed are acquired by a publishing agent configured in the publishing equipment and stored in a memory; the storage information is sent to the coordination service device by the publishing agent.
In the embodiment of the invention, the publishing device is provided with the publishing agent, the subscribing device is provided with the subscribing agent, and devices such as additional message middleware and the like do not need to be added outside the subscribing device and the publishing device, so that the resource consumption of a machine is reduced; the data to be published is stored in the memory through the publishing agent, the stored information is sent to the coordination service device, and the subscription agent subscribes from the memory according to the stored information in the coordination service device, so that the coordination service device is used as an intermediate medium of the publishing agent and the subscription agent, information interaction is performed between the publishing agent and the subscription agent, the publishing and the subscription of the data are completed, transfer of third-party devices such as message intermediate pieces is not needed, data transmission delay is reduced, and data transmission efficiency is improved.
Optionally, when the obtaining module 61 obtains the storage information of the data to be subscribed from the coordination service device in the memory, it is specifically configured to: receiving a subscription task sent by a management device, and determining data information to be subscribed and a subscription mode according to the subscription task, wherein the subscription mode comprises a real-time subscription mode or an offline batch subscription mode; and according to the data information to be subscribed, obtaining the storage information of the data to be subscribed in the memory from the coordination service equipment.
The subscribing module 62 is specifically configured to subscribe the corresponding data from the memory according to the storage information in a subscription manner when subscribing the corresponding data from the memory according to the storage information.
Optionally, the apparatus further includes a reporting module, configured to report a data subscription progress to the management device after subscribing to the corresponding data from the memory according to the storage information.
The data subscription device provided by the embodiment of the invention can execute the data subscription method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
EXAMPLE seven
Fig. 7 is a schematic structural diagram of a publishing device according to a seventh embodiment of the present invention. FIG. 7 illustrates a block diagram of an exemplary publishing device 12, suitable for use in implementing embodiments of the present invention. The publishing device 12 shown in fig. 7 is only an example and should not impose any limitations on the functionality or scope of use of embodiments of the present invention.
As shown in FIG. 7, the publishing device 12 is in the form of a general purpose computing device. The components of publishing device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including the system memory 28 and the processing unit 16.
The system memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM)30 and/or cache memory 32. The publishing device 12 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 34 may be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 7, and commonly referred to as a "hard drive"). Although not shown in FIG. 7, a magnetic disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 by one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the invention.
A program/utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may comprise an implementation of a network environment. Program modules 42 generally carry out the functions and/or methodologies of the described embodiments of the invention.
The publishing device 12 may also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, camera, etc.), with one or more devices that enable a user to interact with the publishing device 12, and/or with any devices (e.g., network card, modem, etc.) that enable the publishing device 12 to communicate with one or more other computing devices. Such communication may be through an input/output (I/O) interface 22. Also, the publishing device 12 may also communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN) and/or a public network, such as the Internet) via the network adapter 20. As shown, the network adapter 20 communicates with other modules of the publishing device 12 over the bus 18. It should be understood that although not shown in the figures, other hardware and/or software modules may be used in conjunction with the publishing device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.
The processing unit 16 executes various functional applications and data processing, for example, implementing a data distribution method provided by an embodiment of the present invention, by executing a program stored in the system memory 28.
The present embodiment further provides a subscribing device, and the structure of the subscribing device refers to fig. 7, which is not described herein again. The difference is that the processing unit 16 in the subscribing device executes various functional applications and data processing, for example, implementing the data subscription method provided by the embodiments of the present invention, by running a program stored in the system memory 28.
Example eight
Fig. 8 is a schematic structural diagram of an eighth data publishing and subscribing system according to an embodiment of the present invention. The method comprises the following steps: the coordination service device 100, the publishing device 200 of the above embodiment and at least one subscribing device 300 of the above embodiment. Fig. 8 shows 2 subscribing devices 300, and each subscribing device 300 can subscribe to data from the memory 400 in a real-time subscription manner or an offline batch subscription manner. The coordination service device 100 is used for receiving and storing the storage information of the data sent by the publishing device 200 in the memory 400; and, transmits the storage information to the subscribing device 300.
Optionally, with continued reference to fig. 8, the memory 400 includes a cache 401 and/or a distributed file system 402; the cache 401 is configured in the publishing device 200; distributed file system 402 is configured on a publishing device or independent of a publishing device.
Optionally, the system further comprises: the management apparatus 500 and the system database 600; the management device 500 is configured to send a publishing task to the publishing device 200, receive a data publishing progress reported by the publishing device 200, and store the data publishing progress in the system database 600; the management device 500 is further configured to send a subscription task to the subscribing device 300, receive a data subscription progress reported by the subscribing device 300, and store the data subscription progress in the system database 600.
Optionally, the system further comprises: a monitoring device 700; for receiving the processing flow progress and the processing state reported by the publishing device 200 and the subscribing device 300.
For the functional description and the effect description of each part in the data publishing and subscribing system, refer to the above embodiments, and are not described herein again.
Example nine
The ninth embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the data publishing method or the data subscribing method provided in any embodiment.
Computer storage media for embodiments of the invention may employ any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + +, or the like, as well as conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments illustrated herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (13)
1. A data distribution method is characterized in that the method is applied to a distribution agent configured in a distribution device,
the method comprises the following steps:
acquiring data to be issued generated by the issuing equipment;
storing the data in a memory;
and sending the storage information of the data in the memory to the coordination service equipment so that a subscription agent configured in the subscription equipment acquires the storage information from the coordination service equipment, and subscribing the corresponding data from the memory according to the storage information.
2. The method according to claim 1, wherein the obtaining of the data to be published generated by the publishing device comprises:
receiving a publishing task sent by a management device, and acquiring data to be published generated by the publishing device according to the publishing task;
after the sending the storage information of the data in the memory to the coordination service device, the method further comprises:
and reporting the data release progress to the management equipment.
3. A data subscription method is applied to a subscription agent configured in a subscription device, and comprises the following steps:
acquiring storage information of data to be subscribed in a memory from coordination service equipment;
according to the storage information, corresponding data are subscribed from a memory;
the data to be subscribed is acquired by a publishing agent configured in publishing equipment and is stored in the memory; and the storage information is sent to the coordination service equipment by the publishing agent.
4. The method according to claim 3, wherein the obtaining of the storage information of the data to be subscribed from the coordination service device in the memory comprises:
receiving a subscription task sent by a management device, and determining data information to be subscribed and a subscription mode according to the subscription task, wherein the subscription mode comprises a real-time subscription mode or an offline batch subscription mode;
according to the data information to be subscribed, obtaining storage information of the data to be subscribed in a memory from the coordination service equipment;
the subscribing the corresponding data from the memory according to the storage information comprises: according to the storage information, corresponding data are subscribed from a memory according to the subscription mode;
after the subscribing the corresponding data from the memory according to the storage information, the method further comprises: and reporting the data subscription progress to the management equipment.
5. A data publishing device applied to a publishing agent configured in a publishing device, comprising:
the acquisition module is used for acquiring the data to be issued generated by the issuing equipment;
the storage module is used for storing the data into a memory;
and the sending module is used for sending the storage information of the data in the memory to the coordination service equipment so that a subscription agent configured in the subscription equipment can acquire the storage information from the coordination service equipment, and subscribe the corresponding data from the memory according to the storage information.
6. A data subscription device is applied to a subscription agent configured in a subscription device, and comprises:
the acquisition module is used for acquiring the storage information of the data to be subscribed in the memory from the coordination service equipment;
the subscription module is used for subscribing corresponding data from the memory according to the storage information;
the data to be subscribed is acquired by a publishing agent configured in publishing equipment and is stored in the memory; and the storage information is sent to the coordination service equipment by the publishing agent.
7. A distribution device, comprising:
one or more processors;
a memory for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the data distribution method of claim 1 or 2.
8. A subscribing device, comprising:
one or more processors;
a memory for storing one or more programs,
the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the data subscription method of claim 3 or 4.
9. A data publish and subscribe system, comprising: a coordinating service device, a publishing device as claimed in claim 7 and at least one subscribing device as claimed in claim 8;
the coordination service equipment is used for receiving and storing the storage information of the data sent by the issuing equipment in the memory; and sending the storage information to the subscribing device.
10. The system of claim 9, wherein the memory comprises a cache and/or a distributed file system;
the cache is configured in the issuing equipment;
the distributed file system is configured on the publishing device or independent of the publishing device.
11. The system of claim 9, further comprising: managing device and system databases;
the management equipment is used for sending a publishing task to the publishing equipment, receiving the data publishing progress reported by the publishing equipment and storing the data publishing progress into the system database;
the management device is further configured to send a subscription task to the subscribing device, receive a data subscription progress reported by the subscribing device, and store the data subscription progress in the system database.
12. The system of claim 9, further comprising: monitoring equipment;
the monitoring device is used for receiving the processing flow progress and the processing state reported by the publishing device and the subscribing device.
13. A computer-readable storage medium on which a computer program is stored, the program, when executed by a processor, implementing the data publishing method as claimed in claim 1 or 2, or the data subscribing method as claimed in claim 3 or 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910390171.2A CN111917814B (en) | 2019-05-10 | 2019-05-10 | Data publishing method, data subscribing method, data publishing device, data subscribing system and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910390171.2A CN111917814B (en) | 2019-05-10 | 2019-05-10 | Data publishing method, data subscribing method, data publishing device, data subscribing system and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111917814A true CN111917814A (en) | 2020-11-10 |
| CN111917814B CN111917814B (en) | 2022-09-06 |
Family
ID=73241750
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910390171.2A Active CN111917814B (en) | 2019-05-10 | 2019-05-10 | Data publishing method, data subscribing method, data publishing device, data subscribing system and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111917814B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113656144A (en) * | 2021-08-17 | 2021-11-16 | 百度在线网络技术（北京）有限公司 | Data publishing system, method and device, electronic equipment and storage medium |
| CN114125057A (en) * | 2022-01-26 | 2022-03-01 | 中科边缘智慧信息科技（苏州）有限公司 | Distributed data interaction method and device for multi-service center |
| WO2024021582A1 (en) * | 2022-07-29 | 2024-02-01 | 中兴通讯股份有限公司 | Network data plane system, data interaction method, and storage medium |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101262494A (en) * | 2008-01-23 | 2008-09-10 | 华为技术有限公司 | Method, client, server and system for processing distributed information |
| CN101511098A (en) * | 2009-02-10 | 2009-08-19 | 中兴通讯股份有限公司 | Distributed net element task management system and method |
| CN102158424A (en) * | 2010-02-02 | 2011-08-17 | 微软公司 | Message transport system using publication and subscription mechanisms |
| CN104052804A (en) * | 2014-06-09 | 2014-09-17 | 深圳先进技术研究院 | Method, device and cluster for sharing data streams between different task topologies |
| US20150006257A1 (en) * | 2013-03-15 | 2015-01-01 | Studio Sbv, Inc. | Subscription-based mobile reading platform |
| CN106228342A (en) * | 2016-07-25 | 2016-12-14 | 四川通发电信股份有限公司 | Configurable type task reporting system based on cloud computing |
| CN106412023A (en) * | 2016-09-05 | 2017-02-15 | 南京臻融软件科技有限公司 | Multi-source data distribution method based on distribution subscription mechanism |
| CN106960054A (en) * | 2017-04-01 | 2017-07-18 | 北京奇虎科技有限公司 | The access method and device of data file |
| CN107624038A (en) * | 2016-07-13 | 2018-01-23 | 北京北方华创微电子装备有限公司 | A kind of monitoring method, micro-electronic machining method and device |
| CN107704537A (en) * | 2017-09-22 | 2018-02-16 | 咪咕动漫有限公司 | A kind of dissemination method, device and the storage medium of object to be updated |
| US20180181636A1 (en) * | 2016-12-27 | 2018-06-28 | Salesforce.Com, Inc. | Org sync async subscribe emails |
| CN108282529A (en) * | 2018-01-23 | 2018-07-13 | 百度在线网络技术（北京）有限公司 | Publication and system, the method and apparatus for subscribing to data |
| CN108287764A (en) * | 2018-01-31 | 2018-07-17 | 上海携程商务有限公司 | Distributed task dispatching method and its system, storage medium, electronic equipment |
| CN109165106A (en) * | 2018-08-24 | 2019-01-08 | 深圳乐信软件技术有限公司 | Method for processing business, device, equipment, storage medium and process flow operation system |
-
2019
- 2019-05-10 CN CN201910390171.2A patent/CN111917814B/en active Active
Patent Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101262494A (en) * | 2008-01-23 | 2008-09-10 | 华为技术有限公司 | Method, client, server and system for processing distributed information |
| CN101511098A (en) * | 2009-02-10 | 2009-08-19 | 中兴通讯股份有限公司 | Distributed net element task management system and method |
| CN102158424A (en) * | 2010-02-02 | 2011-08-17 | 微软公司 | Message transport system using publication and subscription mechanisms |
| US20150006257A1 (en) * | 2013-03-15 | 2015-01-01 | Studio Sbv, Inc. | Subscription-based mobile reading platform |
| CN105210069A (en) * | 2013-03-15 | 2015-12-30 | 谷歌公司 | Subscription-based mobile reading platform |
| CN104052804A (en) * | 2014-06-09 | 2014-09-17 | 深圳先进技术研究院 | Method, device and cluster for sharing data streams between different task topologies |
| CN107624038A (en) * | 2016-07-13 | 2018-01-23 | 北京北方华创微电子装备有限公司 | A kind of monitoring method, micro-electronic machining method and device |
| CN106228342A (en) * | 2016-07-25 | 2016-12-14 | 四川通发电信股份有限公司 | Configurable type task reporting system based on cloud computing |
| CN106412023A (en) * | 2016-09-05 | 2017-02-15 | 南京臻融软件科技有限公司 | Multi-source data distribution method based on distribution subscription mechanism |
| US20180181636A1 (en) * | 2016-12-27 | 2018-06-28 | Salesforce.Com, Inc. | Org sync async subscribe emails |
| CN106960054A (en) * | 2017-04-01 | 2017-07-18 | 北京奇虎科技有限公司 | The access method and device of data file |
| CN107704537A (en) * | 2017-09-22 | 2018-02-16 | 咪咕动漫有限公司 | A kind of dissemination method, device and the storage medium of object to be updated |
| CN108282529A (en) * | 2018-01-23 | 2018-07-13 | 百度在线网络技术（北京）有限公司 | Publication and system, the method and apparatus for subscribing to data |
| CN108287764A (en) * | 2018-01-31 | 2018-07-17 | 上海携程商务有限公司 | Distributed task dispatching method and its system, storage medium, electronic equipment |
| CN109165106A (en) * | 2018-08-24 | 2019-01-08 | 深圳乐信软件技术有限公司 | Method for processing business, device, equipment, storage medium and process flow operation system |
Non-Patent Citations (3)
| Title |
|---|
| XIANGBIN QIU: "A Pubish-Subscribe System for Data Replication and Synchronization Among Intergrated Person-Centric Information Systems", 《DIGITALCOMMONS@USU》 * |
| 李腾: "实时发布/订阅数据分发服务的模型分析", 《无线电工程》 * |
| 高林: "《计算机公共基础—基本知识和使用》", 31 July 2000, 高等教育出版社 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113656144A (en) * | 2021-08-17 | 2021-11-16 | 百度在线网络技术（北京）有限公司 | Data publishing system, method and device, electronic equipment and storage medium |
| CN113656144B (en) * | 2021-08-17 | 2023-08-11 | 百度在线网络技术（北京）有限公司 | Data release system, method and device, electronic equipment and storage medium |
| CN114125057A (en) * | 2022-01-26 | 2022-03-01 | 中科边缘智慧信息科技（苏州）有限公司 | Distributed data interaction method and device for multi-service center |
| CN114125057B (en) * | 2022-01-26 | 2022-04-26 | 中科边缘智慧信息科技（苏州）有限公司 | Distributed data interaction method and device for multi-service center |
| WO2024021582A1 (en) * | 2022-07-29 | 2024-02-01 | 中兴通讯股份有限公司 | Network data plane system, data interaction method, and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111917814B (en) | 2022-09-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107729139B (en) | Method and device for concurrently acquiring resources | |
| CN111831420B (en) | Method for task scheduling, related device and computer program product | |
| CN111917814B (en) | Data publishing method, data subscribing method, data publishing device, data subscribing system and readable storage medium | |
| CN109145023B (en) | Method and apparatus for processing data | |
| CN109033814B (en) | Intelligent contract triggering method, device, equipment and storage medium | |
| CN112579148B (en) | Service message processing method and device based on service agent and electronic equipment | |
| CN111400000A (en) | Network request processing method, device, equipment and storage medium | |
| CN111949402A (en) | Database request processing method and device, computer equipment and storage medium | |
| CN108696559B (en) | Stream processing method and device | |
| CN113760638A (en) | Log service method and device based on kubernets cluster | |
| CN115984022A (en) | Unified account checking method and device for distributed payment system | |
| CN115248735A (en) | Log data output control method, device, equipment and storage medium | |
| CN114356713A (en) | Thread pool monitoring method and device, electronic equipment and storage medium | |
| CN111555894B (en) | Data processing method, device, equipment and storage medium | |
| US10713103B2 (en) | Lightweight application programming interface (API) creation and management | |
| CN115202853A (en) | Virtual reality video cloud computing service method | |
| CN112099933A (en) | Task operation and query method and device, electronic equipment and storage medium | |
| CN110908886A (en) | Data sending method and device, electronic equipment and storage medium | |
| CN112749193A (en) | Workflow processing method and device, storage medium and electronic equipment | |
| CN111917812A (en) | Data transmission control method, device, equipment and storage medium | |
| CN113590437B (en) | Alarm information processing method, device, equipment and medium | |
| CN115334153B (en) | Data processing method and device for service grid | |
| CN113835958B (en) | Distributed task execution progress monitoring method, device, medium and equipment | |
| CN111782426B (en) | Method and device for processing client tasks and electronic equipment | |
| US9747131B1 (en) | System and method for variable aggregation in order for workers in a data processing to share information |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |