CN115134419A - Data transmission method, device, equipment and medium - Google Patents
Data transmission method, device, equipment and medium Download PDFInfo
- Publication number
- CN115134419A CN115134419A CN202210729285.7A CN202210729285A CN115134419A CN 115134419 A CN115134419 A CN 115134419A CN 202210729285 A CN202210729285 A CN 202210729285A CN 115134419 A CN115134419 A CN 115134419A
- Authority
- CN
- China
- Prior art keywords
- data
- message queue
- transmission data
- transmission
- node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000005540 biological transmission Effects 0.000 title claims abstract description 285
- 238000000034 method Methods 0.000 title claims abstract description 61
- 238000004590 computer program Methods 0.000 claims description 19
- 238000012546 transfer Methods 0.000 claims description 4
- 239000000126 substance Substances 0.000 claims 1
- 238000012423 maintenance Methods 0.000 abstract description 4
- 238000012545 processing Methods 0.000 description 14
- 238000007726 management method Methods 0.000 description 13
- 238000010586 diagram Methods 0.000 description 10
- 238000004891 communication Methods 0.000 description 9
- 238000004364 calculation method Methods 0.000 description 8
- 230000006870 function Effects 0.000 description 8
- 238000012544 monitoring process Methods 0.000 description 5
- 238000004458 analytical method Methods 0.000 description 4
- 238000013500 data storage Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 3
- 238000013144 data compression Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000004880 explosion Methods 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000007639 printing Methods 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2457—Query processing with adaptation to user needs
- G06F16/24578—Query processing with adaptation to user needs using ranking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/283—Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention discloses a data transmission method, a data transmission device, data transmission equipment and a data transmission medium. The method is applied to the current node and comprises the following steps: acquiring transmission data of a publishing node and storing the transmission data; and sending the stored transmission data to a corresponding message queue so that a subscribing node subscribing the transmission data can acquire the transmission data, and the publishing node and the current node operate independently. The embodiment of the invention can realize the uniform storage and sending of the transmission data, reduce the maintenance cost of the publishing node, reduce the data transmission cost and improve the efficiency of the subscribing node for acquiring the transmission data.
Description
Technical Field
The present invention relates to the field of big data technologies, and in particular, to a data transmission method, apparatus, device, and medium.
Background
In the era of information explosion, data which can be acquired by people is exponentially multiplied, and the data is acquired by a data acquisition tool through point burying or log printing in a traditional mode and then is sent to a message queue for a business party to use.
The data acquisition tool is used for acquisition, the storage resource of the application terminal needs to be occupied, the application performance of the application terminal is influenced, and the maintenance cost is high.
Disclosure of Invention
The invention provides a data transmission method, a data transmission device, data transmission equipment and a data transmission medium, which are used for realizing the uniform storage and sending of transmission data, reducing the maintenance cost of a publishing node and improving the efficiency of a subscribing node for acquiring the transmission data.
According to an aspect of the present invention, a data transmission method is provided, which is applied to a current node, and the method includes:
acquiring transmission data of a publishing node and storing the transmission data;
and sending the stored transmission data to a corresponding message queue so that a subscribing node subscribing the transmission data can acquire the transmission data, and the publishing node and the current node operate independently.
According to another aspect of the present invention, there is provided a data transmission apparatus, applied in a current node, the apparatus including:
the data acquisition module is used for acquiring transmission data of the publishing node and storing the transmission data;
and the data transmission module is used for sending the stored transmission data to a corresponding message queue so that a subscribing node subscribing the transmission data can acquire the transmission data, and the publishing node and the current node operate independently.
According to another aspect of the present invention, there is provided an electronic apparatus including:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein, the first and the second end of the pipe are connected with each other,
the memory stores a computer program executable by the at least one processor, the computer program being executable by the at least one processor to enable the at least one processor to perform the data transmission method according to any of the embodiments of the invention.
According to another aspect of the present invention, there is provided a computer-readable storage medium storing computer instructions for causing a processor to implement the data transmission method according to any one of the embodiments of the present invention when the computer instructions are executed.
According to another aspect of the present invention, a computer program product is provided, the computer program product comprising a computer program which, when executed by a processor, implements the data transmission method according to any of the embodiments of the present invention.
According to the technical scheme of the embodiment of the invention, the transmission data is stored in the current node by acquiring and storing the transmission data of the publishing node, the transmission data does not need to be stored in the publishing node, the storage resource of the publishing node is saved, the stored transmission data is sent to the corresponding message queue through the current node, so that the transmission data can be acquired by the subscribing node subscribing the data, the transmission link of the transmission data is simplified by storing and sending the transmission data in the current node, the transmission efficiency of the transmission data can be improved, and the transmission cost is reduced.
It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present invention, nor do they necessarily limit the scope of the invention. Other features of the present invention will become apparent from the following description.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a flowchart of a data transmission method according to an embodiment of the present invention;
fig. 2 is a flowchart of a data transmission method according to a second embodiment of the present invention;
fig. 3 is a flowchart of a data transmission method according to a third embodiment of the present invention;
fig. 4a is a system architecture diagram of a data transmission method according to a fourth embodiment of the present invention;
fig. 4b is a data link diagram of a data transmission method according to a fourth embodiment of the present invention;
fig. 5 is a schematic structural diagram of a data transmission apparatus according to a fifth embodiment of the present invention;
fig. 6 is a schematic structural diagram of an electronic device implementing the data transmission method according to the embodiment of the present invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the term "comprises/comprising" and any variations thereof in the description and claims of this invention and the above-described drawings is intended to cover non-exclusive inclusions, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Example one
Fig. 1 is a flowchart of a data transmission method according to an embodiment of the present invention, where the embodiment is applicable to a case of log data transmission, the method may be executed by a data transmission apparatus, the data transmission apparatus may be implemented in a form of hardware and/or software, and the data transmission apparatus may be configured in an electronic device. As shown in fig. 1, the method is applied to a current node, and the method includes:
and S110, acquiring the transmission data of the publishing node and storing the transmission data.
The publishing node refers to a node that generates transmission data. Specifically, the publishing node may be an APP (Application), a device, a web page, or the like, and is not specifically limited in this Application. The number of publishing nodes is at least one. The publishing node may be configured in a K8s (container orchestrator), a virtual machine, or a host. The transmission data refers to data which needs to be collected and transmitted in the data generated by the publishing node. The transmission data may include at least one of device log data, APP log data, and buried point log data. The current node is a node that stores and transmits transmission data generated by the publishing node. In the application, the current node may include a storage module and a data distribution service, the transmission data generated by the publishing node is stored by the storage module, and the data stored in the storage module is sent by the data distribution service. The publishing node may send the transmission data to the current node by means of mounting, or may directly send the transmission data to the current node by means of an API (Application Programming Interface) or an SDK (Software Development Kit).
Specifically, the publishing node directly sends the generated transmission data to the current node, and the publishing node does not store the transmission data and only stores the transmission data by the current node. When the current node stores the transmission data, metadata is generated, wherein the metadata is also called intermediate data or relay data, is data describing the transmission data, is mainly information describing the attribute of the transmission data, and is used for supporting functions such as indicating a storage position, history data, resource searching, file recording and the like.
S120, sending the stored transmission data to a corresponding message queue so that a subscribing node subscribing to the transmission data can obtain the transmission data, and the publishing node and the current node operate independently.
A message queue refers to a container that holds messages during their transmission. Illustratively, the message queue may be a kafka cluster. The number of message queues is at least one. According to the corresponding relation between the transmission data and the message queue, the transmission data can be sent to the corresponding message queue. The corresponding relationship between the transmission data and the message queue may be determined according to configuration information of the transmission data, or according to a geographical location of the subscription node, or according to a storage condition of the message queue. For example, the corresponding relationship between the transmission data and the message queue may be configured for the transmission data, and the transmission data may be sent to the message queue according to the configuration information; or determining a message queue with the same geographical position as the subscribing node in each message queue as a message queue corresponding to the transmission data; and determining the message queue with the least messages stored in each message queue as the message queue corresponding to the transmission data. A subscribing node refers to a node that needs to obtain transmitted data. The number of subscribing nodes is at least one. The subscribing nodes can be divided into different areas according to the geographical positions of the subscribing nodes. The subscribing node needs to subscribe the transmission data to the publishing node, so that the publishing node sends the transmission data to the current node for storage, and sends the transmission data to the message queue through the current node. The transmission data subscribed by each subscription node can be the same or different. Through the message queue, the subscription node can directly acquire the subscribed transmission data, or can perform logic processing on the transmission data through the real-time data processing module, and the subscription node acquires the processed transmission data through the message queue so as to realize consumption of the transmission data. For example, the subscribing node may perform monitoring alarm, intelligent recommendation, or message notification according to the acquired processed transmission data.
Specifically, according to the corresponding relationship between the transmission data and the message queue, the current node sends the transmission data to the corresponding message queue, and the subscription node acquires the transmission data through the message queue. The publishing node and the current node operate independently, or it can be understood that, no matter whether the current node needs to send transmission data, the publishing node sends the transmission data to the current node, so that the transmission data is stored in the current node, if the transmission data subscribed by each subscribing node is stored in the current node, the current node sends the transmission data subscribed by each subscribing node to a corresponding message queue, so that each subscribing node obtains the transmission data, and if the transmission data subscribed by the subscribing node is not stored in the current node, the current node prohibits sending the data to the corresponding message queue.
According to the technical scheme of the embodiment of the invention, the transmission data is stored in the current node by acquiring and storing the transmission data of the publishing node, the transmission data does not need to be stored in the publishing node, the storage resource of the publishing node is saved, the stored transmission data is sent to the corresponding message queue through the current node so that the subscription node subscribing the data can acquire the transmission data, and the transmission link of the transmission data is simplified by storing and sending the transmission data in the current node, so that the flow cost for transmitting the data is reduced, the transmission efficiency of the transmission data can be improved, and the transmission cost is reduced.
On the basis of the above embodiment, the storing the transmission data includes: and storing the transmission data in a storage module, wherein the publishing node is mounted on the storage module.
The storage module refers to a memory used for storing transmission data in the current node. In the storage module, the transmission data can be flexibly stored through the configured storage strategy. For example, the Storage module may transmit data for Storage through at least one of HDFS (Hadoop distributed File System), S3(Simple Storage Service), OSS (Operation Support Systems), COS (object Storage), MINIO (distributed Storage), IPFS (Inter-platform File System, Inter-satellite File System), NAS (Network Attached Storage, Network access Storage), SAS (Serial Attached SCSI), SAN (Storage Area Network), FTP (File Transfer Protocol), SMB (Server Message Block, communication Protocol), and NFS (Network File System).
Specifically, the current node includes a storage module for storing the transmission data. The publishing node is mounted on the storage module, and it can also be understood that the transmission data generated by the publishing node is directly sent to the storage module and stored, and the publishing node does not need to store the transmission data.
The publishing node is mounted in the storage module, the storage module is used for storing the transmission data, the transmission data do not need to be stored in the distribution node, the storage resource of the publishing node is saved, the occurrence probability of the situation that the transmission data are lost due to the fault of the publishing node can be reduced, the safety of the transmission data storage is improved, meanwhile, a plug-in is not needed to be arranged in the publishing node for storing the transmission data, the use performance of the publishing node can be improved, and the maintenance cost for maintaining the publishing node is reduced.
Example two
Fig. 2 is a flowchart of a data transmission method according to a second embodiment of the present invention, where this embodiment sends the stored transmission data to a corresponding message queue on the basis of the foregoing embodiment, and the details are as follows: acquiring a subscription position of the subscription node; detecting a message queue corresponding to the subscription position, and determining the message queue as a message queue corresponding to the transmission data; and sending the transmission data to a corresponding message queue. As shown in fig. 2, the method includes:
s210, acquiring transmission data of the publishing node, and storing the transmission data.
S220, obtaining the subscription position of the subscription node.
The subscription location refers to the geographical location of the subscribing node. Specifically, the current node may obtain the subscription location according to metadata corresponding to the transmission data in the storage module.
And S230, detecting a message queue corresponding to the subscription position, and determining the message queue as a message queue corresponding to the transmission data.
The message queue corresponding to the subscription position refers to a message queue for the subscription node to acquire transmission data. Detecting the message queue corresponding to the subscription position refers to comparing the geographic position of the message queue corresponding to the subscription position with the geographic position of each message queue, and determining the message queue with the same geographic position as the message queue corresponding to the subscription position, wherein the geographic position of the message queue corresponding to the subscription position can be obtained according to the metadata in the storage module.
Specifically, the geographic position of the message queue corresponding to the subscription position is compared with the geographic position of each message queue, and the message queue with the same geographic position as the message queue corresponding to the subscription position is the message queue corresponding to the subscription position, so that the message queue corresponding to the subscription position is determined to be the message queue corresponding to the transmission data. Illustratively, the subscription position is a city a, the geographic position of the message queue 1 is a city b, the geographic position of the message queue 2 is a city a, the geographic position of the message queue corresponding to the subscription position is a city b, and by comparing the geographic position of the message queue corresponding to the subscription position with the geographic position of the message queue 1 and the geographic position of the message queue 2, a message queue with the same geographic position as the message queue corresponding to the subscription position is a message queue 1, and the message queue corresponding to the subscription position is a message queue 1, thereby determining that the message queue corresponding to the transmission data is a message queue 1.
S240, the transmission data are sent to a corresponding message queue, so that a subscribing node subscribing to the transmission data can obtain the transmission data, and the publishing node and the current node operate independently.
Specifically, the current node sends the transmission data to the corresponding message queue according to the corresponding relationship between the transmission data and the message queue. And the subscribing node acquires the transmission data through the message queue.
According to the technical scheme of the embodiment, the subscription position of the subscription node is obtained, the message queue corresponding to the subscription position is determined to be the message queue corresponding to the transmission data, so that the transmission data are sent to the corresponding message queue, the transmission cost of sending the transmission data to the message queue can be reduced, meanwhile, the transmission data are only sent to the message queue corresponding to the subscription position, and the storage resources of the message queue can be saved.
On the basis of the above embodiment, the message queue includes a message queue in the same zone as the subscription position, or a message queue in a different zone from the subscription position.
The message queue in the same region as the subscription position refers to the message queue in the same geographical position as the subscription position. A message queue that is in a different zone from the subscription location refers to a message queue that is in a geographic location that is different from the subscription location. Illustratively, the subscription position is a city, the geographic position of the message queue 1 is b city, and the geographic position of the message queue 2 is a city, so that the message queue in the same region as the subscription position is the message queue 2, and the message queue in a different region from the subscription position is the message queue 1.
Specifically, the message queue corresponding to the subscription position may be a message queue located in the same zone as the subscription position, or may be a message queue located in a different zone from the subscription position.
By dividing the message queue into the message queue in the same area as the subscription position and the message queue in the area different from the subscription position, two different transmission ways can be realized to send the transmission data, and the safety and the flexibility of the transmission data in the transmission process are improved.
On the basis of the above embodiment, the method further comprises the following steps: updating the corresponding relation between the subscription position and the message queue; the detecting a message queue corresponding to the subscription location includes: and determining the message queue corresponding to the subscription position according to the updated corresponding relation between the subscription position and the message queue.
Specifically, in the current node, the corresponding relationship between the subscription position and the message queue may be updated according to transmission data sent by the publishing node, or the corresponding relationship between the subscription position and the message queue may be updated by adjusting metadata in the storage module in a manual configuration manner. Illustratively, data describing the correspondence between the subscription position and the message queue in the transmission data sent by the publishing node for the first time is a, which indicates that the subscription position and the message queue 1 are in the correspondence, data describing the correspondence between the subscription position and the message queue in the transmission data sent by the publishing node for the second time is changed into B, the correspondence between the subscription position and the message queue is updated, and the updated subscription position and the message queue 2 are in the correspondence.
In the prior art, a corresponding relationship between a subscription position and a message queue is configured in each publishing node, and when the corresponding relationship changes, the configuration of each publishing node needs to be updated.
Compared with the prior art, the method and the device have the advantages that the influence of the change of the corresponding relation between the subscription position and the message queue on the publishing node can be reduced, the application performance of the publishing node is improved, and the accuracy of transmitting the transmission data is improved.
EXAMPLE III
Fig. 3 is a flowchart of a data transmission method according to a third embodiment of the present invention, where the present embodiment further includes, on the basis of the foregoing embodiment: and sending the stored transmission data to a data lake, so as to build a bin according to the transmission data through the data lake, and browsing and inquiring the data built in the bin. As shown in fig. 3, the method includes:
s310, acquiring transmission data of the publishing node, and storing the transmission data.
S320, sending the stored transmission data to a corresponding message queue so that a subscribing node subscribing to the transmission data can obtain the transmission data, and the publishing node and the current node operate independently.
S330, sending the stored transmission data to a data lake, so as to build a bin according to the transmission data through the data lake, and browsing and inquiring the data built in the bin.
Data lake refers to a repository or system that stores transmission data in raw format. It stores the transmission data as it is without structuring the data in advance. Binning refers to establishing a data warehouse for the transmission data stored in the data lake. Wherein, the data warehouse at least comprises a horizontal hierarchical data warehouse. For example, the horizontal hierarchical data warehouse may include at least one layer of DWD (data consumer nodes), DIM (Dimension, Dimension layer), DWS (data consumer service), ADS (application data service), and the like.
Specifically, the current node may send the transmission data directly to the data lake, or may send the transmission data to the data lake through the message queue. And the data lake establishes a data warehouse for the transmission data according to the transmission data sent by the current node. Through the data warehouse, the transmission data stored in the data warehouse can be browsed and inquired. For example, APP, applet, large screen or SAAS (Software-as-a-Service) may be used to browse and query the transmitted data stored in the data repository.
According to the technical scheme, the transmission data are sent to the data lake, the transmission data are stored in the data lake, the data storage cost can be reduced, the data storage efficiency is improved, the data lake is used for building the warehouse for the transmission data, the built warehouse is browsed and inquired, the unified management of the transmission data can be achieved, and the management efficiency of the transmission data is improved.
On the basis of the above embodiment, the method further includes: receiving data retransmission information; and acquiring target data corresponding to the data retransmission information, and sending the target data to the message queue again.
The data retransmission information refers to information indicating that the current node retrieves the transmission data from the storage module. The target data refers to transmission data that needs to be retrieved. Specifically, the current node may monitor the message queue, and when transmission data in the message queue is lost or an error occurs, the current node receives data retransmission information generated by the message queue; or, when the transmission data acquired by the subscription node is lost or has an error, the subscription node may generate data retransmission information and send the data retransmission information to the message queue, and the current node receives the data retransmission information by monitoring the message queue. In the present application, the data distribution service may also be configured to monitor the message queue, and in the current node, monitor the message queue through the data distribution service and receive data retransmission information.
Specifically, after receiving the data retransmission information, the current node acquires target data corresponding to the data retransmission information from the storage module according to the data retransmission information, and sends the target data to the message queue.
By receiving the data retransmission information and sending the target data to the message queue again according to the data retransmission information, the retransmission of the target data can be realized, the safety of the target data transmission process is improved, and the subscribing node is ensured to acquire the target data, so that the stability and the safety of the data transmission method are improved.
Example four
Fig. 4a is a system architecture diagram of a data transmission method according to a fourth embodiment of the present invention, and as shown in fig. 4a, the system architecture diagram includes: the system comprises a data access module, a storage module, a data lake and storehouse integrated module and a data calculation and distribution application module.
The data access module is used for mounting the publishing node in the storage module. In the data access module, at least one of POSIX (portable operating System Interface), smb (server Message block), NFS (Network File System), and CSI (Container Storage Interface) may be used to mount the publishing node in the Storage module by a mounting method, or transmit the transmission data generated by the publishing node in the Storage module by SDK or API. The publishing nodes may be configured in containers, hosts, or virtual machines.
The storage module is used for storing transmission data. The transmission data generates metadata in the storage process, and the metadata can be divided into incremental data and full data. The storage module can also be used for performing at least one of copy management, universe life cycle management and control, data encryption, hierarchical storage, file level data storage management, file schema (mode) management, file metadata management, data life cycle management, data security management and control, data disaster recovery, data compression archiving, data sharing and the like on the transmission data. In the storage module, at least one method of HDFS, S3, OSS, COS, MINIO, IPFS, NAS, SAS, SAN, FTP, SMB, NFS and the like can be used for storing transmission data, so that the storage diversity of the transmission data is improved, and the performance and the safety of the publishing node are improved.
The data lake storage integrated module is used for storing the transmission data in the storage module through the data lake and the data bin. The data lake in the data lake storage integrated module can also perform multidimensional analysis on the transmission data. The multidimensional analysis may include at least one of incremental computation, full computation, scientific computation, data snapshot, security management and the like.
The data calculation and distribution application module is used for realizing the functions of data distribution, data calculation, data service, data warehouse building and the like on the transmission data. In the data calculation and distribution application module, data distribution can be realized through a data distribution service and a message queue, data calculation and data warehouse building are realized through a real-time data processing module, and a unified query module provides data transmission services such as data query and browsing for at least one of APP, small programs, large screens or SAAS. The data calculation and distribution application module also comprises service applications, namely subscription nodes, and is used for realizing functions of monitoring alarm, intelligent recommendation, message notification and the like.
In a specific example, fig. 4b is a data link diagram of a data transmission method according to a fourth embodiment of the present invention, as shown in fig. 4 b:
the data access module uses POSIX to mount a distribution node configured in the host or virtual machine into the storage module, and uses CSI to mount a distribution node configured in the container (k8s) into the storage module. And the publishing node generates transmission data and directly stores the transmission data into the storage module. The transmission data may be at least one of an APP print log, a device print log, and a buried point print log.
The storage module stores the transmission data by methods of S3, COS, MINIO and the like, forms metadata, and performs management such as copy management, data encryption, data compression archiving and the like.
The data distribution service in the data calculation and distribution application module sends the transmission data in the storage module to a message queue according to the subscription position of the subscription node, wherein the message queue comprises a same-region message queue (same-region data-kafka cluster) or a cross-region message queue (same-region data-kafka cluster), the same-region message queue comprises local kafka, and the cross-region message queue comprises a region a kafka, a region b kafka, a region c kafka and the like.
Optionally, the data distribution service may also monitor a storage condition of transmission data in the message queue, receive data retransmission information fed back by the message queue when the transmission data in the message queue is lost or has an error, acquire target data corresponding to the data retransmission information from the storage module, and send the target data to the message queue.
The message queue can directly send the transmission data subscribed by the subscription node to the subscription node, so that the subscription node realizes at least one of functions of monitoring alarm, intelligent recommendation, message notification and the like.
Optionally, the message queue may further send the transmission data to the real-time data processing module, and the real-time data processing module implements processing of the transmission data through the flink computing engine, and sends the processed transmission data to the subscription node through the message queue, so that the subscription node implements at least one of functions of monitoring and warning, intelligent recommendation, message notification, and the like.
Optionally, the real-time data processing module may further establish a data bin according to the transmission data, and store the transmission data into the data bin. Specifically, the Data Warehouse comprises three layers, the first layer comprises DWD (Data WareHouse services) and DIM (Dimension layer), the second layer comprises DWS (Data WareHouse service), the third layer comprises ADS (application Data service), and the real-time processing service realizes the storage of the transmission Data according to the function of each layer of Data Warehouse.
Optionally, the real-time data processing module may further directly obtain the transmission data through the data lake, and store the transmission data in the data lake in the data bin.
Optionally, the data lake may directly obtain the transmission data through the storage module, store the transmission data in the storage module in the data lake, and perform multidimensional analysis on the obtained data; and transmission data can be acquired from the message queue through a flink acquisition module, stored in a data lake and subjected to multidimensional analysis.
Optionally, the unified query module in the data calculation and distribution application module may query data in a data lake or a data warehouse, so as to provide data services such as data transmission query and browsing for at least one of APP, small program, large screen, or SAAS.
According to the technical scheme of the embodiment of the invention, the transmission data is stored in the current node by acquiring and storing the transmission data of the publishing node, the transmission data does not need to be stored in the publishing node, the storage resource of the publishing node is saved, the stored transmission data is sent to the corresponding message queue through the current node so that the subscription node subscribing the data can acquire the transmission data, and the transmission link of the transmission data is simplified by storing and sending the transmission data in the current node, so that the flow cost for transmitting the data is reduced, the transmission efficiency of the transmission data can be improved, and the transmission cost is reduced.
EXAMPLE five
Fig. 5 is a schematic structural diagram of a data transmission device according to a fifth embodiment of the present invention. As shown in fig. 5, the apparatus is applied to a current node, and includes: a data acquisition module 501 and a data transmission module.
The data acquiring module 501 is configured to acquire transmission data of a publishing node and store the transmission data;
a data transmission module 502, configured to send the stored transmission data to a corresponding message queue, so that a subscribing node subscribing to the transmission data obtains the transmission data, and the publishing node and the current node operate independently.
According to the technical scheme of the embodiment of the invention, the transmission data is stored in the current node by acquiring and storing the transmission data of the publishing node, the transmission data does not need to be stored in the publishing node, the storage resource of the publishing node is saved, the stored transmission data is sent to the corresponding message queue through the current node, so that the transmission data can be acquired by the subscribing node subscribing the data, and the transmission link of the transmission data is simplified by storing and sending the transmission data in the current node, so that the flow cost used by the transmission data is reduced, the transmission efficiency of the transmission data can be improved, and the transmission cost is reduced.
Optionally, the data obtaining module 501 is specifically configured to:
and storing the transmission data in a storage module, wherein the publishing node is mounted on the storage module.
Optionally, the data transmission module 502 includes:
a subscription position acquisition unit, configured to acquire a subscription position of the subscription node;
a message queue determining unit, configured to detect a message queue corresponding to the subscription position, and determine the message queue as a message queue corresponding to the transmission data;
and the data transmission unit is used for sending the transmission data to the corresponding message queue.
Optionally, the message queue includes a message queue located in the same zone as the subscription position, or a message queue located in a different zone from the subscription position.
Optionally, the data transmission module 502 further includes:
the message queue updating unit updates the corresponding relation between the subscription position and the message queue;
the message queue determining unit is specifically configured to:
and determining the message queue corresponding to the subscription position according to the updated corresponding relation between the subscription position and the message queue.
Optionally, the apparatus further comprises:
and the data warehousing module is used for sending the stored transmission data to a data lake so as to perform warehousing according to the transmission data through the data lake and browse and query the warehoused data.
Optionally, the apparatus further comprises:
the retransmission information receiving module is used for receiving data retransmission information;
and the target data sending module is used for acquiring the target data corresponding to the data retransmission information and sending the target data to the message queue again.
The data transmission device provided by the embodiment of the invention can execute the data transmission method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
EXAMPLE six
FIG. 6 illustrates a block diagram of an electronic device 10 that may be used to implement an embodiment of the invention. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.
As shown in fig. 6, the electronic device 10 includes at least one processor 11, and a memory communicatively connected to the at least one processor 11, such as a Read Only Memory (ROM)12, a Random Access Memory (RAM)13, and the like, wherein the memory stores a computer program executable by the at least one processor, and the processor 11 can perform various suitable actions and processes according to the computer program stored in the Read Only Memory (ROM)12 or the computer program loaded from a storage unit 18 into the Random Access Memory (RAM) 13. In the RAM 13, various programs and data necessary for the operation of the electronic apparatus 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other via a bus 14. An input/output (I/O) interface 15 is also connected to bus 14.
A number of components in the electronic device 10 are connected to the I/O interface 15, including: an input unit 16 such as a keyboard, a mouse, or the like; an output unit 17 such as various types of displays, speakers, and the like; a storage unit 18 such as a magnetic disk, optical disk, or the like; and a communication unit 19 such as a network card, modem, wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.
The processor 11 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, or the like. The processor 11 performs the various methods and processes described above, such as the data transmission method.
In some embodiments, the data transfer method may be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 10 via the ROM 12 and/or the communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the data transmission method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the data transfer method by any other suitable means (e.g., by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), system on a chip (SOCs), Complex Programmable Logic Devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
Computer programs for implementing the methods of the present invention can be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be performed. A computer program can execute entirely on a machine, partly on a machine, as a stand-alone software package partly on a machine and partly on a remote machine or entirely on a remote machine or server.
In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the electronic device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), blockchain networks, and the Internet.
The computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The Server may be a cloud Server, which is also called a cloud computing Server or a cloud host, and is a host product in a cloud computing service system, so as to solve the defects of high management difficulty and weak service expansibility in the conventional physical host and VPS (Virtual Private Server) service.
It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present invention may be executed in parallel, sequentially, or in different orders, and are not limited herein as long as the desired results of the technical solution of the present invention can be achieved.
The above-described embodiments should not be construed as limiting the scope of the invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A data transmission method is applied to a current node and comprises the following steps:
acquiring transmission data of a publishing node and storing the transmission data;
and sending the stored transmission data to a corresponding message queue so that a subscribing node subscribing the transmission data can acquire the transmission data, and the publishing node and the current node operate independently.
2. The method of claim 1, wherein the storing the transmission data comprises:
and storing the transmission data in a storage module, wherein the publishing node is mounted on the storage module.
3. The method of claim 1, wherein sending the stored transmission data to a corresponding message queue comprises:
acquiring a subscription position of the subscription node;
detecting a message queue corresponding to the subscription position, and determining the message queue as a message queue corresponding to the transmission data;
and sending the transmission data to a corresponding message queue.
4. The method of claim 3, wherein the message queue comprises a message queue in a same zone as the subscription location or a message queue in a different zone from the subscription location.
5. The method of claim 3, further comprising:
updating the corresponding relation between the subscription position and the message queue;
the detecting a message queue corresponding to the subscription location includes:
and determining the message queue corresponding to the subscription position according to the updated corresponding relation between the subscription position and the message queue.
6. The method of claim 1, further comprising:
and sending the stored transmission data to a data lake, so as to build a bin according to the transmission data through the data lake, and browsing and inquiring the data built in the bin.
7. The method of claim 1, further comprising:
receiving data retransmission information;
and acquiring target data corresponding to the data retransmission information, and sending the target data to the message queue again.
8. A data transmission apparatus, applied in a current node, comprising:
the data acquisition module is used for acquiring transmission data of the publishing node and storing the transmission data;
and the data transmission module is used for sending the stored transmission data to a corresponding message queue so that a subscribing node subscribing the transmission data can acquire the transmission data, and the publishing node and the current node operate independently.
9. An electronic device, characterized in that the electronic device comprises:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the data transfer method of any one of claims 1-7.
10. A computer-readable storage medium storing computer instructions for causing a processor to perform the data transmission method of any one of claims 1 to 7 when executed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210729285.7A CN115134419A (en) | 2022-06-24 | 2022-06-24 | Data transmission method, device, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210729285.7A CN115134419A (en) | 2022-06-24 | 2022-06-24 | Data transmission method, device, equipment and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115134419A true CN115134419A (en) | 2022-09-30 |
Family
ID=83379429
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210729285.7A Pending CN115134419A (en) | 2022-06-24 | 2022-06-24 | Data transmission method, device, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115134419A (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108762953A (en) * | 2018-05-25 | 2018-11-06 | 连云港杰瑞电子有限公司 | A kind of message queue implementation method |
| CN109032534A (en) * | 2018-08-30 | 2018-12-18 | 百度在线网络技术（北京）有限公司 | A kind of method for message transmission, device, equipment and medium |
| CN110718084A (en) * | 2019-10-10 | 2020-01-21 | 腾讯科技（深圳）有限公司 | Driving information transmission method, device, system, equipment and storage medium |
| CN111459686A (en) * | 2020-03-17 | 2020-07-28 | 无锡华云数据技术服务有限公司 | Queue message storing and forwarding method and system and computer device with operating system |
| CN111787055A (en) * | 2020-05-22 | 2020-10-16 | 中国科学院信息工程研究所 | Redis-based transaction mechanism and multi-data center oriented data distribution method and system |
| CN112702190A (en) * | 2020-12-11 | 2021-04-23 | 广东电力通信科技有限公司 | Regional alarm message pushing method and system based on message queue |
| CN113301558A (en) * | 2020-06-29 | 2021-08-24 | 阿里巴巴集团控股有限公司 | Message transmission method, device, system and storage medium |
| CN113905089A (en) * | 2021-09-06 | 2022-01-07 | 浪潮软件股份有限公司 | Multi-user message pushing method and system based on publish-subscribe mode |
| CN114338801A (en) * | 2022-02-23 | 2022-04-12 | 国网湖南省电力有限公司 | Lightweight publishing and subscribing method and system suitable for intelligent measuring equipment, equipment and storage medium |
-
2022
- 2022-06-24 CN CN202210729285.7A patent/CN115134419A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108762953A (en) * | 2018-05-25 | 2018-11-06 | 连云港杰瑞电子有限公司 | A kind of message queue implementation method |
| CN109032534A (en) * | 2018-08-30 | 2018-12-18 | 百度在线网络技术（北京）有限公司 | A kind of method for message transmission, device, equipment and medium |
| CN110718084A (en) * | 2019-10-10 | 2020-01-21 | 腾讯科技（深圳）有限公司 | Driving information transmission method, device, system, equipment and storage medium |
| CN111459686A (en) * | 2020-03-17 | 2020-07-28 | 无锡华云数据技术服务有限公司 | Queue message storing and forwarding method and system and computer device with operating system |
| CN111787055A (en) * | 2020-05-22 | 2020-10-16 | 中国科学院信息工程研究所 | Redis-based transaction mechanism and multi-data center oriented data distribution method and system |
| CN113301558A (en) * | 2020-06-29 | 2021-08-24 | 阿里巴巴集团控股有限公司 | Message transmission method, device, system and storage medium |
| CN112702190A (en) * | 2020-12-11 | 2021-04-23 | 广东电力通信科技有限公司 | Regional alarm message pushing method and system based on message queue |
| CN113905089A (en) * | 2021-09-06 | 2022-01-07 | 浪潮软件股份有限公司 | Multi-user message pushing method and system based on publish-subscribe mode |
| CN114338801A (en) * | 2022-02-23 | 2022-04-12 | 国网湖南省电力有限公司 | Lightweight publishing and subscribing method and system suitable for intelligent measuring equipment, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110147398B (en) | Data processing method, device, medium and electronic equipment | |
| US10560465B2 (en) | Real time anomaly detection for data streams | |
| US11836533B2 (en) | Automated reconfiguration of real time data stream processing | |
| US10447772B2 (en) | Managed function execution for processing data streams in real time | |
| US10169709B2 (en) | Avoiding incompatibility between data and computing processes to enhance computer performance | |
| CA2896865C (en) | Method and system for using a recursive event listener on a node in hierarchical data structure | |
| CN111917687B (en) | Method and device for circularly pushing reminding message | |
| CN110837423A (en) | Method and device for automatically acquiring data of guided transport vehicle | |
| CN113377626B (en) | Visual unified alarm method, device, equipment and medium based on service tree | |
| US10331484B2 (en) | Distributed data platform resource allocator | |
| CN116366692A (en) | High-performance intelligent edge terminal system | |
| CN114860782B (en) | Data query method, device, equipment and medium | |
| US11954012B2 (en) | Client-side telemetry data filter model | |
| CN115134419A (en) | Data transmission method, device, equipment and medium | |
| CN115514618A (en) | Alarm event processing method and device, electronic equipment and medium | |
| CN111294231B (en) | Resource management method and system | |
| US20160162559A1 (en) | System and method for providing instant query | |
| CN112433891A (en) | Data processing method and device and server | |
| CN109388546B (en) | Method, device and system for processing faults of application program | |
| CN111858260A (en) | Information display method, device, equipment and medium | |
| JP2017535847A (en) | Method and system for content caching and transmission | |
| CN112749204A (en) | Method and device for reading data | |
| CN115203139A (en) | Log query method and device, electronic equipment and storage medium | |
| CN115665240A (en) | Proxy service adding method, device, electronic equipment and storage medium | |
| CN118170532A (en) | Processing method, processing system and storage medium for data of Internet of things |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |