CN111917812A - Data transmission control method, device, equipment and storage medium - Google Patents
Data transmission control method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN111917812A CN111917812A CN201910388811.6A CN201910388811A CN111917812A CN 111917812 A CN111917812 A CN 111917812A CN 201910388811 A CN201910388811 A CN 201910388811A CN 111917812 A CN111917812 A CN 111917812A
- Authority
- CN
- China
- Prior art keywords
- transmission
- configuration information
- target
- target node
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/565—Conversion or adaptation of application format or content
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0876—Network utilisation, e.g. volume of load or congestion level
- H04L43/0894—Packet rate
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Environmental & Geological Engineering (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
- Computer And Data Communications (AREA)
Abstract
The embodiment of the invention discloses a data transmission control method, a data transmission control device, data transmission control equipment and a storage medium. The method comprises the following steps: splitting the data source in the acquired transmission configuration information to obtain a splitting result matched with the parallel processing quantity; generating local transmission configuration information corresponding to the splitting result according to a target source and a local calculation operator in the transmission configuration information, and distributing the local transmission configuration information to target agent equipment in the agent equipment set; and receiving transmission process information of each target node in the transmission stream reported by the target agent equipment, and monitoring the transmission stream and/or the real-time running state of the target node according to the transmission process information. The technical scheme of the embodiment of the invention can further strengthen the supervision and control on the transmission flow and the nodes in the data transmission process, and realize the dynamic adjustment on the transmission flow or the transmission control mode of the target node in the data transmission process so as to further improve the data transmission efficiency.
Description
Technical Field
The present invention relates to data transmission control technologies, and in particular, to a data transmission control method, apparatus, device, and storage medium.
Background
In the internet industry, the era of "big data" has come, and companies and organizations are all researching and constructing their own big data processing platforms to dig out various useful information. The large-scale data are all acquired from various application systems, and for large-scale companies, the application systems are numerous, and the data generation, storage modes and specifications are different, especially the log data of the application systems.
Therefore, how to construct an effective log transmission system to reliably transmit large-scale log data to a big data platform in real time for data processing is an important problem to be solved at present.
Disclosure of Invention
The embodiment of the invention provides a data transmission control method, a data transmission control device, data transmission control equipment and a data storage medium, which are used for distributing data to a plurality of nodes in a transmission stream for parallel transmission and strengthening supervision and control on the transmission stream and the nodes.
In a first aspect, an embodiment of the present invention provides a data transmission control method, including:
acquiring transmission configuration information, and splitting a data source in the transmission configuration information to obtain a splitting result matched with a set parallel processing quantity;
according to a target source and a local calculation operator in the transmission configuration information, generating local transmission configuration information corresponding to each split result respectively, and distributing the local transmission configuration information to at least one target agent device in an agent device set;
each piece of local transmission configuration information is used for indicating each target agent device to operate at least one target node respectively, and generating a transmission stream matched with the transmission configuration information in a combined mode for data transmission;
and receiving the transmission process information of each target node in the transmission stream reported by the at least one target agent device, and monitoring the transmission stream and/or the real-time running state of the target node according to the transmission process information.
In a second aspect, an embodiment of the present invention further provides a data transmission control apparatus, including:
the data source splitting module is used for acquiring transmission configuration information and splitting the data source in the transmission configuration information to obtain a splitting result matched with the set parallel processing quantity;
a local transmission configuration information distribution module, configured to generate, according to a destination source and a local computation operator in the transmission configuration information, local transmission configuration information corresponding to each of the split results, and distribute the local transmission configuration information to at least one target proxy device in a proxy device set;
each piece of local transmission configuration information is used for indicating each target agent device to operate at least one target node respectively, and generating a transmission stream matched with the transmission configuration information in a combined mode for data transmission;
and the real-time running state monitoring module is used for receiving the transmission process information of each target node in the transmission stream periodically reported by the at least one target agent device and monitoring the transmission stream and/or the real-time running state of the target node according to the transmission process information.
In a third aspect, an embodiment of the present invention further provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the data transmission control method according to any one of the embodiments of the present invention when executing the program.
In a fourth aspect, the embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements the data transmission control method according to any one of the embodiments of the present invention.
The embodiment of the invention provides a data transmission control method, a data transmission control device, data transmission control equipment and a storage medium, wherein a data source in transmission configuration information is split, local transmission configuration information corresponding to each split result is generated and distributed to target proxy equipment in a proxy equipment set, each target proxy equipment is indicated to operate a corresponding target node, and a transport stream is generated in a combined mode to carry out data transmission; according to the transmission progress information of each target node in the transmission stream reported by the target agent device, the transmission stream and/or the real-time operation state of the target node are monitored, the agent devices in the agent device set are uniformly allocated, the transmission stream formed by a plurality of target nodes can be quickly generated to realize parallel transmission of data sources, the information is obtained in real time in the transmission of each target node, the supervision and control on the transmission stream and the nodes can be further strengthened in the data transmission process, the transmission control mode of the transmission stream or the target node is dynamically adjusted in the data transmission process, and the data transmission efficiency is further improved.
Drawings
Fig. 1 is a flowchart of a data transmission control method according to a first embodiment of the present invention;
fig. 2 is a flowchart of a data transmission control method according to a second embodiment of the present invention;
fig. 3 is a flowchart of a data transmission control method according to a third embodiment of the present invention;
fig. 4 is a configuration diagram of a data transmission control apparatus according to a fourth embodiment of the present invention;
fig. 5 is a schematic structural diagram of a computer device in the fifth embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
It should be further noted that, for the convenience of description, only some but not all of the relevant aspects of the present invention are shown in the drawings. Before discussing exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although a flowchart may describe the operations (or steps) as a sequential process, many of the operations can be performed in parallel, concurrently or simultaneously. In addition, the order of the operations may be re-arranged. The process may be terminated when its operations are completed, but may have additional steps not included in the figure. The processes may correspond to methods, functions, procedures, subroutines, and the like.
In order to facilitate understanding of the technical solutions of the embodiments of the present invention, first, the implementation mechanisms of the embodiments of the present invention are briefly described. The technical scheme of the embodiment of the invention mainly relates to the following hardware equipment: the system comprises a plurality of data source storage locations, a plurality of destination source storage locations, a management device (typically, implemented by a cluster), and a proxy device set, wherein the proxy device set comprises a plurality of proxy devices. The method of each embodiment of the invention is mainly completed by the management equipment.
When a data source (typically, log data) stored in a set storage location needs to be backed up or transmitted to a destination source storage location, a piece of transmission configuration information may be sent to the management device, where the transmission configuration information indicates the data source that needs to be transmitted, the destination source storage location to which the data source needs to be transmitted, and a data processing mode of the data source during transmission. The management device divides the data source into small units of data of a set parallel processing number based on the transmission configuration information to perform parallel transmission of the data source. In a specific example, the data source contains 1 thousand pieces of data, and if the pre-specified parallel processing amount is 10, the data source can be divided into 10 small units of 1 thousand pieces of data. Then, the management device may select one or more proxy devices from the multiple proxy device sets, and send the corresponding number of small unit data, so that the proxy devices start corresponding processes (i.e., nodes) according to the number of the received small unit data, and respectively transmit each small unit data to the corresponding destination source storage location. In a specific example, the management device allocates 3 small unit data to a selected one of the proxy devices, and the proxy device starts three threads (nodes) to transmit the three small unit data to the destination source storage location, respectively, after receiving the small unit data. It should be noted that the set of nodes that are turned on when the data source is transmitted together form a transmission stream.
Example one
Fig. 1 is a flowchart of a data transmission control method according to an embodiment of the present invention, where this embodiment is applicable to a case where a data source is transmitted to a destination source storage location in parallel through one or more proxy devices, and the method may be executed by a data transmission control apparatus according to an embodiment of the present invention, where the apparatus may be implemented in a software and/or hardware manner, and the apparatus may be integrated into a server or a server cluster, and the server or the server cluster is used as a management device and is used in cooperation with one or more proxy devices in a proxy device set. As shown in fig. 1, the method of this embodiment specifically includes:
s110, acquiring transmission configuration information, and splitting a data source in the transmission configuration information to obtain a splitting result matched with the set parallel processing quantity.
As mentioned above, the transmission configuration information is the configuration information sent to the management device. The transmission configuration information may include: data source information, destination source information and local calculation operator information.
The data source information is used for specifying a storage location of the data source to be transmitted, and the management device can acquire the data source to be transmitted through the data source information. Typically, the data source information may include: a log cluster path or a log path. Meanwhile, the data format of the data source may be: logs, files, secondary files, or various types of database files, etc.
The destination source information is used to specify a destination source storage location to which the data source is transmitted, and typically, the destination source information may include: hdfs (Hadoop Distributed File System) path, BigPipe (redesigned basic dynamic Web service architecture) path, kafka (an open source streaming platform developed by the Apache software Foundation) path, File path, and the like.
The local calculation operator information is used to specify a processing mode or a data conversion mode of the node for the data source to be transmitted, for example, in a process that the data source backs up from the data source storage location to the destination source storage location, in order to adapt to a storage form of the data source at the destination source storage location, data format conversion needs to be performed on the data source, and a specific data format conversion mode is determined by the local calculation operator.
The transmission configuration information is mainly configured in a configuration interface pushed to a user by the management equipment, and the user can flexibly select data source information, target source information and local calculation operator information according to actual needs.
After the management device obtains the transmission configuration information, the data source in the transmission configuration information may be split to obtain a split result matching the set parallel processing number. The splitting processing may be splitting of data or splitting of a data set, and the specific splitting operation is to divide a data source with a large data amount into a plurality of local data sources with a small data amount, so as to implement parallel data processing.
As described above, the data source needs to be split according to the preset number of parallel processes to obtain multiple split results (i.e., the small unit data). The parallel processing amount may be preset according to an actual load condition, or a data amount included in the data source, or may be directly specified by a user, which is not limited in this embodiment.
S120, according to the destination source and the local calculation operator in the transmission configuration information, generating local transmission configuration information corresponding to each split result respectively, and distributing the local transmission configuration information to at least one target agent device in an agent device set.
The local transmission configuration information is used for indicating each target agent device to operate at least one target node respectively, and combining and generating a transmission stream matched with the transmission configuration information for data transmission.
In this embodiment, each local transmission configuration information corresponds to one splitting result, and the local transmission configuration information includes: and the corresponding splitting result comprises data source information, destination source information and local calculation operator information of part of data sources.
Each target node starts a process corresponding to the quantity value as a node according to the quantity value of the received local transmission configuration information, and correspondingly allocates one local transmission configuration information to each node for processing, and each node can correspondingly establish three processes (processes) according to data source information, destination source information and local calculation operator information of partial data sources included in the allocated local transmission configuration information, and respectively executes: 1. reading a part of data sources according to the data source information; 2. performing data conversion on the read data source according to the local calculation operator; and 3, writing the data after the data conversion into a destination source storage position corresponding to the destination source information.
In this embodiment, it is necessary to establish a node matching with the quantitative value direction of the local transmission configuration information in the at least one target node, and perform parallel data transmission processing on all data sources corresponding to the transmission configuration information, where the nodes together form a transport stream for transmitting all data sources.
The number of the local transmission configuration information allocated to each target node may be the same or different. Specifically, the amount of the local transmission configuration information allocated by each node may be determined according to the actual load of each target node, or all the local transmission configuration information may be directly and equally distributed to each target node, which is not limited in this embodiment.
S130, receiving transmission process information of each target node in the transmission stream reported by the at least one target agent device, and monitoring the transmission stream and/or the real-time running state of the target node according to the transmission process information.
In this embodiment, in order to enhance effective monitoring on the transport stream and each target node, it is necessary to control each target agent device to report the transmission process information of each node in the transport stream in real time.
Typically, each target agent device may be specified to report the transmission process information of the target node started by itself periodically every set time duration, for example, 5 minutes or 10 minutes, or the management device may send a reporting instruction to each target agent device every set time duration, so that each target agent device reports the transmission process information of the target node started by itself according to the received reporting instruction.
For example, if 1000 data are allocated to one target node for transmission, and the target node has successfully transmitted 100 data to the destination source storage location at the current time point, the transmission process information reported by the target node is 100/1000-10% at the current time point.
The real-time operating status refers to information for describing a transmission status of a target node or a transport stream to a data source, and the real-time operating status may include: normal running state, blocking state, slow running state, pause state, abnormal interrupt state, etc.
In this embodiment, the real-time operating state of each target node or the whole transport stream may be monitored according to the transmission process information reported by the target node of the target agent device at different times. For example, if a target node keeps 10% of the transmission progress information in 10 minutes, it may be determined that the target node is currently in a blocked state, and if all target nodes or more than 80% of the target nodes in the transmission stream are within 10 minutes, and the increment of the transmission progress information does not exceed 2%, it may be determined that the transmission stream is currently in a blocked state, and so on.
In this embodiment, based on the real-time monitoring of the transmission progress information of each target node in the transmission stream, the real-time operating state of each target node or the entire transmission stream can be dynamically obtained, and thus the transmission state of the target node or the transmission stream can be quickly and accurately adjusted, so as to improve the transmission efficiency of the data source.
The embodiment of the invention provides a data transmission control method, which comprises the steps of splitting a data source in transmission configuration information, generating local transmission configuration information corresponding to each splitting result, distributing the local transmission configuration information to target proxy equipment in a proxy equipment set, indicating each target proxy equipment to operate a corresponding target node, and generating a transport stream in a combined manner to carry out data transmission; according to the transmission progress information of each target node in the transmission stream reported by the target agent device, the transmission stream and/or the real-time operation state of the target node are monitored, the agent devices in the agent device set are uniformly allocated, the transmission stream formed by a plurality of target nodes can be quickly generated to realize parallel transmission of data sources, the information is obtained in real time in the transmission of each target node, the supervision and control on the transmission stream and the nodes can be further strengthened in the data transmission process, the transmission control mode of the transmission stream or the target node is dynamically adjusted in the data transmission process, and the data transmission efficiency is further improved.
On the basis of the foregoing embodiments, after receiving the transmission progress information of each target node in the transmission stream reported by the at least one target agent device, the method may further include: updating and storing the transmission process information of each target node; and the stored transmission process information is used for carrying out retrospective retransmission or breakpoint continuous transmission on the data which is transmitted and completed by each target node in the restarted transmission stream after the transmission stream is abnormally interrupted.
In this embodiment, the management device may further update and store the received transmission process information of each target node, where the update and storage refers to storing the latest transmission process information sent by each target node after receiving the transmission process information reported by each target node, for example, when 5% of the transmission process information first reported by the target node a at 2019.04.26.11:00:00 is stored, first storing the 5% of the transmission process information, and when 30% of the transmission process information reported by the target node a at 2019.04.26.11:05:00 is stored, updating and storing the current transmission process information of the target node to 30%.
The advantages of such an arrangement are: when the transport stream is abnormally interrupted, in the restarted transport stream, the data which is transmitted and completed by each target node can be subjected to backtracking retransmission or breakpoint continuous transmission, so that the data which is transmitted and completed is prevented from being transmitted again, the transmission efficiency of the data is further improved, and the transmission time of the data is shortened.
Wherein, the backtracking retransmission means backtracking to the latest data transmission position, and continuing to transmit subsequent data by taking the position as a starting point; breakpoint resume refers to the transfer of data from where the file was last interrupted. The two modes can ensure that the data is not retransmitted from the initial position, but a new starting point of data transmission can be selected according to the current transmission progress of the target node, and further, the transmission of repeated data can be reduced after an abnormal condition occurs.
On the basis of the above embodiments, the local computation operator may be componentized configuration information; wherein the local computation operator may include: custom processing components uploaded by the user, and/or standard processing components configured by default by the system.
In this embodiment, in order to further improve the flexibility of the functions of the management device and achieve the componentized (plug-in) management of the transmission configuration information, a user can add a custom processing component into the transmission configuration information in a plug-in adding manner, so as to add a new data processing function which can meet the user requirements, and further, the new data transmission mode can be conveniently added and adapted to the heterogeneous data transmission mode.
Of course, it can be understood by those skilled in the art that, in addition to the above-mentioned local computation operator, the data source may also be componentized configuration information, so that a user can self-define introduction of an unstructured data source, and accordingly, when the user needs to set the transmission configuration information, only a self-defined data source configuration component is added to the configuration page where the configuration information is transmitted (a main implementation manner may be that writing code inherits a parent class provided in advance by the system to obtain a new class as the data source configuration component), and a self-defined data source is specified based on the data source component, so as to further enrich the generality and extensibility of the management device of the embodiment of the present invention.
Example two
Fig. 2 is a flowchart of a data transmission control method according to a second embodiment of the present invention, where this embodiment is optimized based on the above embodiment, and in this embodiment, the monitoring of the real-time running state of the transport stream and/or the target node according to the transmission progress information is implemented as: respectively counting transmission process information of each target node in the transmission stream under at least two monitoring time points; calculating the transmission rate and the transmission completion degree respectively corresponding to each target node according to the statistical result of the transmission process information; determining the real-time running state of each target node according to the transmission rate and the transmission completion degree; and carrying out statistical calculation on the real-time running state of each target node, and taking the statistical calculation result as the real-time running state of the transport stream.
Correspondingly, the method of the embodiment of the invention comprises the following steps:
s210, acquiring transmission configuration information, and splitting a data source in the transmission configuration information to obtain a splitting result matched with the set parallel processing quantity.
S220, according to the destination source and the local calculation operator in the transmission configuration information, generating local transmission configuration information corresponding to each split result respectively, and distributing the local transmission configuration information to at least one target agent device in an agent device set.
The local transmission configuration information is used for indicating each target agent device to operate at least one target node respectively, and combining and generating a transmission stream matched with the transmission configuration information for data transmission.
S230, receiving transmission process information of each target node in the transmission stream reported by the at least one target agent device.
S240, respectively counting transmission progress information of each target node in the transmission stream under at least two monitoring time points.
The monitoring time point specifically refers to a time point when the management device receives the transmission process information of the target node reported by the agent device.
In this embodiment, by obtaining the transmission process information of each target node reported by the proxy device at each monitoring time point, the transmission process information of each target node at two or more consecutive monitoring time points can be summarized.
And S250, calculating the transmission rate and the transmission completion degree respectively corresponding to each target node according to the statistical result of the transmission process information.
According to the data transmission increment of each node at two or more continuous time points and the time interval determined by the two or more time points, the transmission rate of each target node can be determined, and the latest transmission progress information of each target node can be used as the transmission completion degree of each target node.
For example, if the transmission process information reported by the target node a at 2019.04.26.11:00:00 is 5%, and the transmission process information reported by the target node a at 2019.04.26.11:05:00 is 30%, it may be determined that the target node a has transmitted 25% of the data amount in 5 minutes, and according to the total data amount of the target node, for example: 5M, the data transmission increment in 5 minutes can be determined to be 1M, and then the transmission rate of the target node a can be determined to be 1/300 (M/s).
And S260, determining the real-time running state of each target node according to the transmission rate and the transmission completion degree.
S270, performing statistical calculation on the real-time running state of each target node, and taking the statistical calculation result as the real-time running state of the transport stream.
In this embodiment, a statistical calculation rule may be preset, and based on the statistical calculation rule, the real-time operation state of each target node is statistically calculated to obtain a statistical calculation result as the real-time operation state of the transport stream. For example, the number of target nodes whose transmission speeds are equal to or greater than a first speed threshold, the number of target nodes located between the first speed and a second speed, and the number of target nodes equal to or less than the second speed are counted, respectively, wherein the first speed is greater than the second speed.
Alternatively, the transmission speed (average of the transmission speeds of the target nodes) and the transmission completion (average of the transmission completion of each target node) of the entire transmission stream may be calculated based on the transmission speed and the transmission completion of each target node as the statistical calculation result.
Of course, those skilled in the art may also perform statistical calculation on the real-time operating status of each of the target nodes in other manners to obtain corresponding statistical calculation results to measure the real-time operating status of the transport stream.
The embodiment of the invention provides a data transmission control method, which comprises the steps of splitting a data source in transmission configuration information, generating local transmission configuration information corresponding to each splitting result, distributing the local transmission configuration information to target proxy equipment in a proxy equipment set, indicating each target proxy equipment to operate a corresponding target node, and generating a transport stream in a combined manner to carry out data transmission; according to the transmission progress information of each target node in the transmission stream reported by the target agent device, the transmission stream and/or the real-time operation state of the target node are monitored, the agent devices in the agent device set are uniformly allocated, the transmission stream formed by a plurality of target nodes can be quickly generated to realize parallel transmission of data sources, the information is obtained in real time in the transmission of each target node, the supervision and control on the transmission stream and the nodes can be further strengthened in the data transmission process, the transmission control mode of the transmission stream or the target node is dynamically adjusted in the data transmission process, and the data transmission efficiency is further improved.
On the basis of the foregoing embodiments, before distributing the local transmission configuration information to at least one target proxy device in the proxy device set, the method may further include:
and selecting the at least one target proxy device in the proxy device set according to the number of the operated nodes of each proxy device in the proxy device set and the adopted load balancing algorithm.
On the basis of the above embodiments, the data transmission control method may be implemented by a server cluster formed by at least two standard servers;
the server cluster is used for selecting a matched standard server to process the acquired transmission configuration information according to the total number of transmission streams currently controlled by each standard server, the number of currently controlled proxy devices, the priority weight of the transmission streams and an adopted load balancing algorithm.
The advantage of such an arrangement is that each standard server and each target agent device can be deployed in a distributed manner in a cluster manner to construct a large-scale data transmission system.
EXAMPLE III
Fig. 3 is a flowchart of a data transmission control method according to a third embodiment of the present invention, which is optimized based on the third embodiment. In this embodiment, after distributing the local transmission configuration information to at least one target proxy device in the proxy device set, the method further specifically includes: receiving node state information of each target node reported by the at least one target agent device;
correspondingly, after monitoring the real-time operating state of the transport stream and/or the target node according to the transmission process information, the method further includes: and selecting a matched transmission control mode according to the transmission stream and/or the real-time operation state of each target node and the node state information of each target node, and performing transmission control on the transmission stream and/or the target nodes.
Correspondingly, the method of the embodiment may include:
s310, acquiring transmission configuration information, and splitting a data source in the transmission configuration information to obtain a splitting result matched with the set parallel processing quantity.
S320, according to the destination source and the local calculation operator in the transmission configuration information, generating local transmission configuration information corresponding to each split result respectively, and distributing the local transmission configuration information to at least one target agent device in an agent device set.
The local transmission configuration information is used for indicating each target agent device to operate at least one target node respectively, and combining and generating a transmission stream matched with the transmission configuration information for data transmission.
S330, receiving the transmission progress information of each target node in the transmission stream reported by the at least one target agent device.
S340, receiving the node state information of each target node reported by the at least one target agent device.
And S350, respectively counting transmission process information of each target node in the transmission stream under at least two monitoring time points.
And S360, calculating the transmission rate and the transmission completion degree respectively corresponding to each target node according to the statistical result of the transmission process information.
And S370, determining the real-time running state of each target node according to the transmission rate and the transmission completion degree.
And S380, performing statistical calculation on the real-time running state of each target node, and taking the statistical calculation result as the real-time running state of the transport stream.
And S390, selecting a matched transmission control mode according to the transport stream, and/or the real-time operation state of each target node, and the node state information of each target node, and performing transmission control on the transport stream and/or the target nodes.
In an optional implementation manner of this embodiment, the transmission control manner for the transport stream includes at least one of the following: restart, pause, delete, and modify;
the transmission control mode for the target node comprises at least one of the following: restart, pause, and restart after pause.
In this embodiment, a corresponding transmission control manner may be selected to perform transmission control on the transport stream or the target node according to the real-time operating state of the transport stream or the target node.
In a specific example, if a target node is currently in a blocking state, the target node may be restarted, or the target node may be first suspended and restarted after being suspended after a period of time, and the purpose of performing the transmission control is to improve the current blocking state of the target node.
On the basis of the foregoing embodiments, the transmission configuration information may further include: an open option for at least one additional function;
correspondingly, after the obtaining of the transmission configuration information, the method may further include: acquiring a target accessory function selected to be started in the transmission configuration information; and starting a service program matched with the target additional function to provide the target additional function in the data transmission process.
The advantages of such an arrangement are: a user customizes a task frame of the user according to actual data uploading requirements, and customized content (target attachment function) can be used as an independent component and independently run in the agent equipment.
Optionally, the additional functionality may include at least one of: a notification function, an alarm function, a self-check function, and an integrity detection function.
The notification function specifically means that after a transport stream completes data, a system where a destination source is located is notified that the transmission is finished, and the notification mode mainly includes two modes, one mode is real-time (a downstream interface is called), and the other mode is delayed (an identification file is issued, and the destination source automatically scans a document); the alarm function specifically refers to that the management equipment scans the running state of the agent equipment, monitors the processing pressure of the agent equipment and sends an alarm signal to the corresponding agent equipment when the agent equipment is found to be abnormal in processing; the self-checking function is to provide a log query mode, specifically, the management device reads a log interface of the agent device, and can check the activity state of the agent device reported by the agent device in the management device.
Example four
Fig. 4 is a schematic structural diagram of a data transmission control apparatus according to a fourth embodiment of the present invention, and as shown in fig. 4, the apparatus includes: a data source splitting module 410, a local transmission configuration information distribution module 420, and a real-time running state monitoring module 430. Wherein:
the data source splitting module 410 is configured to obtain transmission configuration information, and split the data source in the transmission configuration information to obtain a splitting result that matches the set parallel processing quantity.
A local transmission configuration information distribution module 420, configured to generate, according to a destination source and a local computation operator in the transmission configuration information, local transmission configuration information corresponding to each of the split results, and distribute the local transmission configuration information to at least one target proxy device in the proxy device set.
Each piece of local transmission configuration information is used for indicating each target agent device to operate at least one target node respectively, and generating a transmission stream matched with the transmission configuration information in a combined mode for data transmission;
a real-time operation status monitoring module 430, configured to receive transmission process information of each target node in the transmission stream periodically reported by the at least one target agent device, and monitor the transmission stream and/or a real-time operation status of the target node according to the transmission process information.
The embodiment of the invention provides a data transmission control device, which splits a data source in transmission configuration information, generates local transmission configuration information corresponding to each split result, distributes the local transmission configuration information to target proxy equipment in a proxy equipment set, indicates each target proxy equipment to operate a corresponding target node, and generates a transport stream in a combined manner to perform data transmission; according to the transmission progress information of each target node in the transmission stream reported by the target agent device, the transmission stream and/or the real-time operation state of the target node are monitored, the agent devices in the agent device set are uniformly allocated, the transmission stream formed by a plurality of target nodes can be quickly generated to realize parallel transmission of data sources, the information is obtained in real time in the transmission of each target node, the supervision and control on the transmission stream and the nodes can be further strengthened in the data transmission process, the transmission control mode of the transmission stream or the target node is dynamically adjusted in the data transmission process, and the data transmission efficiency is further improved.
On the basis of the foregoing embodiments, the real-time operation status monitoring module 430 may be specifically configured to:
respectively counting transmission process information of each target node in the transmission stream under at least two monitoring time points;
calculating the transmission rate and the transmission completion degree respectively corresponding to each target node according to the statistical result of the transmission process information;
determining the real-time running state of each target node according to the transmission rate and the transmission completion degree;
and carrying out statistical calculation on the real-time running state of each target node, and taking the statistical calculation result as the real-time running state of the transport stream.
On the basis of the above embodiments, the method may further include:
a node state information receiving module, configured to receive node state information of each target node reported by at least one target agent device after distributing the local transmission configuration information to the at least one target agent device in the agent device set;
correspondingly, the method can further comprise the following steps: and the transmission control module is used for selecting a matched transmission control mode according to the transmission stream, and/or the real-time operation state of each target node and the node state information of each target node after monitoring the transmission stream and/or the real-time operation state of the target node according to the transmission process information, and performing transmission control on the transmission stream and/or the target node.
On the basis of the foregoing embodiments, the transmission control manner for the transport stream may include at least one of the following: restart, pause, delete, and modify; and/or
The transmission control manner for the target node may include at least one of: restart, pause, and restart after pause.
On the basis of the above embodiments, the method may further include:
the progress information storage module is used for updating and storing the transmission progress information of each target node after receiving the transmission progress information of each target node in the transmission stream reported by the at least one target agent device;
and the stored transmission process information is used for carrying out retrospective retransmission or breakpoint continuous transmission on the data which is transmitted and completed by each target node in the restarted transmission stream after the transmission stream is abnormally interrupted.
On the basis of the above embodiments, the method may further include:
and the target agent device selection module is used for selecting at least one target agent device in the agent device set according to the number of the running nodes of each agent device in the agent device set and the adopted load balancing algorithm before distributing the local transmission configuration information to the at least one target agent device in the agent device set.
On the basis of the above embodiments, the local computation operator may be componentized configuration information;
wherein the local computation operator may include: custom processing components uploaded by the user, and/or standard processing components configured by default by the system.
On the basis of the foregoing embodiments, the transmission configuration information may further include: an open option for at least one additional function;
correspondingly, the method can further comprise the following steps: the system comprises an additional function starting module, a transmission configuration information acquiring module and a control module, wherein the additional function starting module is used for acquiring a target additional function which is selected to be started and is included in the transmission configuration information after acquiring the transmission configuration information;
and starting a service program matched with the target additional function to provide the target additional function in the data transmission process.
On the basis of the above embodiments, the additional functions may include at least one of the following: a notification function, an alarm function, a self-check function, and an integrity detection function.
On the basis of the above embodiments, the data transmission control device is implemented by a server cluster composed of at least two standard servers;
the server cluster is used for selecting a matched standard server to process the acquired transmission configuration information according to the total number of transmission streams currently controlled by each standard server, the number of currently controlled proxy devices, the priority weight of the transmission streams and an adopted load balancing algorithm.
The data transmission control device can execute the data transmission control method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the executed data transmission control method.
EXAMPLE five
Fig. 5 is a schematic structural diagram of a computer device according to a fifth embodiment of the present invention. FIG. 5 illustrates a block diagram of an exemplary computer device 12 suitable for use in implementing embodiments of the present invention. The computer device 12 shown in FIG. 5 is only an example and should not bring any limitations to the functionality or scope of use of embodiments of the present invention.
As shown in FIG. 5, computer device 12 is in the form of a general purpose computing device. The components of computer device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including the system memory 28 and the processing unit 16.
The system memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM)30 and/or cache memory 32. Computer device 12 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 34 may be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 5, and commonly referred to as a "hard drive"). Although not shown in FIG. 5, a magnetic disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 by one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the invention.
A program/utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may comprise an implementation of a network environment. Program modules 42 generally carry out the functions and/or methodologies of the described embodiments of the invention.
The processing unit 16 executes various functional applications and data processing by executing programs stored in the system memory 28, for example, to implement a data transmission control method provided by an embodiment of the present invention.
Namely: the processing unit implements, when executing the program: acquiring transmission configuration information, and splitting a data source in the transmission configuration information to obtain a splitting result matched with a set parallel processing quantity;
according to a target source and a local calculation operator in the transmission configuration information, generating local transmission configuration information corresponding to each split result respectively, and distributing the local transmission configuration information to at least one target agent device in an agent device set;
each piece of local transmission configuration information is used for indicating each target agent device to operate at least one target node respectively, and generating a transmission stream matched with the transmission configuration information in a combined mode for data transmission;
and receiving the transmission process information of each target node in the transmission stream reported by the at least one target agent device, and monitoring the transmission stream and/or the real-time running state of the target node according to the transmission process information.
EXAMPLE six
A sixth embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements a data transmission control method according to any of the embodiments of the present invention:
namely: the program when executed by a processor implements: acquiring transmission configuration information, and splitting a data source in the transmission configuration information to obtain a splitting result matched with a set parallel processing quantity;
according to a target source and a local calculation operator in the transmission configuration information, generating local transmission configuration information corresponding to each split result respectively, and distributing the local transmission configuration information to at least one target agent device in an agent device set;
each piece of local transmission configuration information is used for indicating each target agent device to operate at least one target node respectively, and generating a transmission stream matched with the transmission configuration information in a combined mode for data transmission;
and receiving the transmission process information of each target node in the transmission stream reported by the at least one target agent device, and monitoring the transmission stream and/or the real-time running state of the target node according to the transmission process information.
Any combination of one or more computer-readable media may be employed. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (13)
1. A method for controlling data transmission, comprising:
acquiring transmission configuration information, and splitting a data source in the transmission configuration information to obtain a splitting result matched with a set parallel processing quantity;
according to a target source and a local calculation operator in the transmission configuration information, generating local transmission configuration information corresponding to each split result respectively, and distributing the local transmission configuration information to at least one target agent device in an agent device set;
each piece of local transmission configuration information is used for indicating each target agent device to operate at least one target node respectively, and generating a transmission stream matched with the transmission configuration information in a combined mode for data transmission;
and receiving the transmission process information of each target node in the transmission stream reported by the at least one target agent device, and monitoring the transmission stream and/or the real-time running state of the target node according to the transmission process information.
2. The method of claim 1, wherein monitoring the transport stream and/or the real-time operation status of the target node according to the transmission progress information comprises:
respectively counting transmission process information of each target node in the transmission stream under at least two monitoring time points;
calculating the transmission rate and the transmission completion degree respectively corresponding to each target node according to the statistical result of the transmission process information;
determining the real-time running state of each target node according to the transmission rate and the transmission completion degree;
and carrying out statistical calculation on the real-time running state of each target node, and taking the statistical calculation result as the real-time running state of the transport stream.
3. The method of claim 1, further comprising, after distributing the local transfer configuration information to at least one target proxy device in a set of proxy devices:
receiving node state information of each target node reported by the at least one target agent device;
after monitoring the real-time operation state of the transport stream and/or the target node according to the transmission process information, the method further comprises the following steps:
and selecting a matched transmission control mode according to the transmission stream and/or the real-time operation state of each target node and the node state information of each target node, and performing transmission control on the transmission stream and/or the target nodes.
4. The method of claim 3, wherein:
the transmission control mode for the transmission stream comprises at least one of the following: restart, pause, delete, and modify; and/or
The transmission control mode for the target node comprises at least one of the following: restart, pause, and restart after pause.
5. The method of claim 1, further comprising, after receiving transmission progress information of each target node in the transmission stream reported by the at least one target agent device:
updating and storing the transmission process information of each target node;
and the stored transmission process information is used for carrying out retrospective retransmission or breakpoint continuous transmission on the data which is transmitted and completed by each target node in the restarted transmission stream after the transmission stream is abnormally interrupted.
6. The method of claim 1, further comprising, prior to distributing the local transfer configuration information to at least one target proxy device in a set of proxy devices:
and selecting the at least one target proxy device in the proxy device set according to the number of the operated nodes of each proxy device in the proxy device set and the adopted load balancing algorithm.
7. The method of any of claims 1-6, wherein the local computation operator is a componentized configuration information;
wherein the local computation operator comprises: custom processing components uploaded by the user, and/or standard processing components configured by default by the system.
8. The method according to any one of claims 1-6, wherein the transmitting the configuration information further comprises: an open option for at least one additional function;
after acquiring the transmission configuration information, the method further comprises:
acquiring a target accessory function selected to be started in the transmission configuration information;
and starting a service program matched with the target additional function to provide the target additional function in the data transmission process.
9. The method of claim 8, wherein the additional functionality comprises at least one of: a notification function, an alarm function, a self-check function, and an integrity detection function.
10. The method according to any one of claims 1-6, wherein:
the data transmission control method is realized by a server cluster formed by at least two standard servers;
the server cluster is used for selecting a matched standard server to process the acquired transmission configuration information according to the total number of transmission streams currently controlled by each standard server, the number of currently controlled proxy devices, the priority weight of the transmission streams and an adopted load balancing algorithm.
11. An apparatus for controlling data transmission, comprising:
the data source splitting module is used for acquiring transmission configuration information and splitting the data source in the transmission configuration information to obtain a splitting result matched with the set parallel processing quantity;
a local transmission configuration information distribution module, configured to generate, according to a destination source and a local computation operator in the transmission configuration information, local transmission configuration information corresponding to each of the split results, and distribute the local transmission configuration information to at least one target proxy device in a proxy device set;
each piece of local transmission configuration information is used for indicating each target agent device to operate at least one target node respectively, and generating a transmission stream matched with the transmission configuration information in a combined mode for data transmission;
and the real-time running state monitoring module is used for receiving the transmission process information of each target node in the transmission stream periodically reported by the at least one target agent device and monitoring the transmission stream and/or the real-time running state of the target node according to the transmission process information.
12. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method according to any of claims 1-10 when executing the program.
13. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1-10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910388811.6A CN111917812B (en) | 2019-05-10 | 2019-05-10 | Data transmission control method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910388811.6A CN111917812B (en) | 2019-05-10 | 2019-05-10 | Data transmission control method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111917812A true CN111917812A (en) | 2020-11-10 |
| CN111917812B CN111917812B (en) | 2022-09-30 |
Family
ID=73242595
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910388811.6A Active CN111917812B (en) | 2019-05-10 | 2019-05-10 | Data transmission control method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111917812B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113810502A (en) * | 2021-09-27 | 2021-12-17 | 远峰科技股份有限公司 | Communication data transmission method, system, computer equipment and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070089039A1 (en) * | 2005-09-30 | 2007-04-19 | Kabushiki Kaisha Toshiba | Data transfer control unit and data transfer control method |
| CN104618236A (en) * | 2015-01-21 | 2015-05-13 | 网宿科技股份有限公司 | Parallel data transmission system and method of acceleration network |
| US20150215945A1 (en) * | 2014-01-28 | 2015-07-30 | Mediatek Inc. | Buffer Status Report and Logical Channel Prioritization for Dual Connectivity |
| CN106713944A (en) * | 2016-12-30 | 2017-05-24 | 北京奇虎科技有限公司 | Method and apparatus for processing streaming data task |
| CN106850188A (en) * | 2017-01-24 | 2017-06-13 | 中国航天系统科学与工程研究院 | A kind of data transmission system based on multichannel isomery one-way transmission path |
| CN107623646A (en) * | 2017-09-06 | 2018-01-23 | 华为技术有限公司 | Data flow transmission method, send equipment and receiving device |
-
2019
- 2019-05-10 CN CN201910388811.6A patent/CN111917812B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070089039A1 (en) * | 2005-09-30 | 2007-04-19 | Kabushiki Kaisha Toshiba | Data transfer control unit and data transfer control method |
| US20150215945A1 (en) * | 2014-01-28 | 2015-07-30 | Mediatek Inc. | Buffer Status Report and Logical Channel Prioritization for Dual Connectivity |
| CN104618236A (en) * | 2015-01-21 | 2015-05-13 | 网宿科技股份有限公司 | Parallel data transmission system and method of acceleration network |
| CN106713944A (en) * | 2016-12-30 | 2017-05-24 | 北京奇虎科技有限公司 | Method and apparatus for processing streaming data task |
| CN106850188A (en) * | 2017-01-24 | 2017-06-13 | 中国航天系统科学与工程研究院 | A kind of data transmission system based on multichannel isomery one-way transmission path |
| CN107623646A (en) * | 2017-09-06 | 2018-01-23 | 华为技术有限公司 | Data flow transmission method, send equipment and receiving device |
Non-Patent Citations (2)
| Title |
|---|
| JINQI ZHU: "A Mobility Prediction-based Adaptive Data Gathering", 《GLOBAL TELECOMMUNICATIONS CONFERENCE》 * |
| 姜春茂: "一种基于P2P的并行传输模型", 《计算机科学》 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113810502A (en) * | 2021-09-27 | 2021-12-17 | 远峰科技股份有限公司 | Communication data transmission method, system, computer equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111917812B (en) | 2022-09-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2021129367A1 (en) | Method and apparatus for monitoring distributed storage system | |
| KR101888029B1 (en) | Method and system for monitoring virtual machine cluster | |
| US8868984B2 (en) | Relevant alert delivery in a distributed processing system with event listeners and alert listeners | |
| US8954811B2 (en) | Administering incident pools for incident analysis | |
| CN109117252B (en) | Method and system for task processing based on container and container cluster management system | |
| CN107729138B (en) | Method and device for analyzing high-performance distributed vector space data | |
| CN112579304A (en) | Resource scheduling method, device, equipment and medium based on distributed platform | |
| CN111526049B (en) | Operation and maintenance system, operation and maintenance method, electronic device and storage medium | |
| CN109542781B (en) | Block chain consensus algorithm testing method and device, calculating device and storage medium | |
| CN110717132A (en) | Data collection method and pushing method for full-link monitoring system and related equipment | |
| US9766995B2 (en) | Self-spawning probe in a distributed computing environment | |
| US10296431B2 (en) | Generation of debugging log list in a blade server environment | |
| US9686174B2 (en) | Scalable extendable probe for monitoring host devices | |
| CN111917812B (en) | Data transmission control method, device, equipment and storage medium | |
| US10122602B1 (en) | Distributed system infrastructure testing | |
| CN110930110B (en) | Distributed flow monitoring method and device, storage medium and electronic equipment | |
| CN111162938A (en) | Data processing system and method | |
| Imran et al. | Cloud-niagara: A high availability and low overhead fault tolerance middleware for the cloud | |
| CN111130882A (en) | Monitoring system and method of network equipment | |
| CN111343152B (en) | Data processing method and device, electronic equipment and storage medium | |
| CN113656239A (en) | Monitoring method and device for middleware and computer program product | |
| CN109088929B (en) | Method and device for sending information | |
| CN111581059A (en) | Spark application monitoring method, system, equipment and storage medium | |
| CN112445597A (en) | Timing task scheduling method and device | |
| CN112860504A (en) | Monitoring method and device, computer storage medium and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |