CN114363411A - Data transmission method, device, system and computer storage medium - Google Patents
Data transmission method, device, system and computer storage medium Download PDFInfo
- Publication number
- CN114363411A CN114363411A CN202111631053.XA CN202111631053A CN114363411A CN 114363411 A CN114363411 A CN 114363411A CN 202111631053 A CN202111631053 A CN 202111631053A CN 114363411 A CN114363411 A CN 114363411A
- Authority
- CN
- China
- Prior art keywords
- upstream
- data transmission
- equipment
- data
- upstream equipment
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000005540 biological transmission Effects 0.000 title claims abstract description 205
- 238000000034 method Methods 0.000 title claims abstract description 63
- 238000011144 upstream manufacturing Methods 0.000 claims abstract description 353
- 230000002776 aggregation Effects 0.000 claims abstract description 16
- 238000004220 aggregation Methods 0.000 claims abstract description 16
- 238000013507 mapping Methods 0.000 claims description 10
- 230000000007 visual effect Effects 0.000 claims description 5
- 238000012546 transfer Methods 0.000 description 12
- 238000010586 diagram Methods 0.000 description 7
- 238000004590 computer program Methods 0.000 description 2
- 230000007547 defect Effects 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 239000003086 colorant Substances 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 238000013024 troubleshooting Methods 0.000 description 1
Images
Abstract
The invention discloses a data transmission method, data transmission equipment, a data transmission system and a computer storage medium. The method comprises the following steps: generating a hierarchical relationship configuration file of a plurality of upstream devices; for any upstream equipment, judging whether the upstream equipment has parent upstream equipment or not according to the hierarchical relationship configuration file, if so, transmitting data corresponding to the upstream equipment to the parent upstream equipment of the upstream equipment, and if not, transmitting the data corresponding to the upstream equipment to the summarizing equipment; the aggregation device receives data transmitted by an upstream device that does not have a parent upstream device. By adopting the scheme, the chaos of the data transmission process can be avoided, and the fault tracing can be rapidly carried out when a certain node breaks down.
Description
Technical Field
The present invention relates to the field of data transmission technologies, and in particular, to a data transmission method, device, system, and computer storage medium.
Background
In some large business services, there are typically scenarios where multiple upstream devices transmit data to the aggregation device. The data transmission mode adopted in the scene is as follows: each upstream device transmits its own data to the aggregation device. As shown in fig. 1, the upstream devices A, B, C, D, E, F, G each transmit data to the aggregation device S.
However, the inventor finds that the following defects exist in the prior art in the implementation process: when the number of the upstream devices is large, confusion of data transmitted to the summarizing device can be caused, and a fault node cannot be located quickly when data transmission fails.
Disclosure of Invention
In view of the above, the present invention has been made to provide a data transmission method, device, system and computer storage medium that overcome the above problems or at least partially solve the above problems.
According to a first aspect of the present invention, there is provided a data transmission method, which is applied to data transmission from a plurality of upstream devices having a hierarchical relationship to a summarizing device, and includes:
generating a hierarchical relationship configuration file of the plurality of upstream devices;
for any upstream device, judging whether the upstream device has a parent upstream device or not according to the hierarchical relationship configuration file, if so, transmitting data corresponding to the upstream device to the parent upstream device of the upstream device, and if not, transmitting data corresponding to the upstream device to the summarizing device;
the aggregation equipment receives data transmitted by upstream equipment without parent upstream equipment.
In an optional embodiment, the method further comprises: for any upstream equipment, judging whether the upstream equipment has sub-level upstream equipment or not according to the hierarchical relation configuration file;
if so, merging the data transmitted by the sub-level upstream equipment of the upstream equipment and the data stored by the upstream equipment, and taking the merged data as the data corresponding to the upstream equipment;
and if not, taking the data stored by the upstream equipment as the data corresponding to the upstream equipment.
In an optional implementation manner, the transmitting the data corresponding to the upstream device further includes: if the data transmission is successful, generating feedback information of the successful data transmission;
and if the data transmission fails, generating feedback information of the data transmission failure.
In an optional embodiment, the method further comprises:
displaying the data transmission state of each upstream device in a visual interface;
wherein the data transmission state comprises one of the following states: wait for transmission, in data transmission, data transmission success, and data transmission failure.
In an optional implementation manner, the transmitting the data corresponding to the upstream device further includes:
and calling a data transmission middleware through a preset interface so that the data transmission middleware can transmit data corresponding to the upstream equipment.
In an optional implementation manner, the transmitting, by the data transmission middleware, data corresponding to the upstream device further includes:
and the data transmission middleware divides a task for transmitting the data corresponding to the upstream equipment into a plurality of subtasks and distributes the subtasks to a plurality of concurrent working threads for execution.
In an alternative embodiment, said distributing said plurality of subtasks to a plurality of concurrent worker thread executions further comprises:
dividing the plurality of subtasks into a plurality of subtask groups according to the number of the working threads; wherein the number of the subtask groups is consistent with the number of the working threads;
and establishing a one-to-one mapping relation between the subtask group and the working thread, and distributing the subtasks in the subtask group to the working thread having the mapping relation with the subtask group for execution.
In an optional embodiment, the method further comprises: responding to the data transmission time configuration operation of a user, and generating a data transmission time configuration file;
the determining, for any upstream device, whether the upstream device has a parent upstream device according to the hierarchical relationship configuration file further includes: and any upstream equipment determines transmission starting time according to the data transmission time configuration file, and executes the operation of judging whether the upstream equipment has parent-level upstream equipment or not according to the hierarchical relationship configuration file at the transmission starting time.
According to a second aspect of the present invention, there is provided a data transmission method performed by any one of a plurality of upstream apparatuses having a hierarchical relationship; the method comprises the following steps:
judging whether the upstream equipment has parent-level upstream equipment or not according to the generated hierarchical relationship configuration file;
if so, transmitting the data corresponding to the upstream equipment to the parent upstream equipment of the upstream equipment;
and if not, transmitting the data corresponding to the upstream equipment to the summarizing equipment.
In an optional embodiment, the method further comprises:
judging whether the upstream equipment has sublevel upstream equipment or not according to the hierarchical relation configuration file;
if so, merging the data transmitted by the subordinate upstream equipment of the upstream equipment and the data stored by the upstream equipment, and taking the merged data as the data corresponding to the upstream equipment;
and if not, taking the data stored in the upstream equipment as the data corresponding to the upstream equipment.
In an optional implementation manner, the transmitting the data corresponding to the upstream device further includes: if the data transmission is successful, generating feedback information of the successful data transmission;
and if the data transmission fails, generating feedback information of the data transmission failure.
In an optional implementation manner, the transmitting the data corresponding to the upstream device further includes:
and calling a data transmission middleware through a preset interface so that the data transmission middleware transmits data corresponding to the upstream equipment.
In an optional embodiment, the method further comprises:
and determining transmission starting time according to the generated data transmission time configuration file, and executing the operation of judging whether the upstream equipment has parent-level upstream equipment according to the generated hierarchical relationship configuration file at the transmission starting time.
According to a third aspect of the present invention, there is provided an upstream apparatus comprising:
the judging module is used for judging whether the upstream equipment has parent-level upstream equipment or not according to the generated hierarchical relationship configuration file;
the transmission module is used for transmitting the data corresponding to the upstream equipment to the parent upstream equipment of the upstream equipment if the judgment result of the judgment module is positive; and if the judgment result of the judgment module is negative, transmitting the data corresponding to the upstream equipment to the summarizing equipment.
According to a fourth aspect of the present invention, there is provided a data transmission system comprising: a plurality of the upstream devices, the aggregation device, and the configuration module;
the configuration module is used for generating a hierarchical relationship configuration file of the plurality of upstream devices;
the summarizing device is used for receiving data transmitted by the upstream device without the parent-level upstream device.
According to a fifth aspect of the present invention, a computer storage medium is provided, where at least one executable instruction is stored in the storage medium, and the executable instruction causes a processor to execute operations corresponding to the data transmission method.
The invention discloses a data transmission method, a device, a system and a computer storage medium, wherein the data transmission method comprises the following steps: generating a hierarchical relationship configuration file of a plurality of upstream devices; for any upstream equipment, judging whether the upstream equipment has parent upstream equipment or not according to the hierarchical relationship configuration file, if so, transmitting data corresponding to the upstream equipment to the parent upstream equipment of the upstream equipment, and if not, transmitting the data corresponding to the upstream equipment to the summarizing equipment; the aggregation device receives data transmitted by an upstream device that does not have a parent upstream device. By adopting the scheme, the chaos of the data transmission process can be avoided, and the fault tracing can be rapidly carried out when a certain node breaks down.
The foregoing description is only an overview of the technical solutions of the present invention, and the embodiments of the present invention are described below in order to make the technical means of the present invention more clearly understood and to make the above and other objects, features, and advantages of the present invention more clearly understandable.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:
FIG. 1 illustrates a data transmission diagram of the prior art;
fig. 2 is a schematic flowchart illustrating a data transmission method according to an embodiment of the present invention;
fig. 3 is a data transmission diagram of a data transmission middleware according to an embodiment of the present invention;
fig. 4 is a schematic diagram illustrating data transmission according to an embodiment of the present invention;
fig. 5 is a flowchart illustrating a data transmission method according to a second embodiment of the present invention;
FIG. 6 is a schematic structural diagram of an upstream apparatus according to a third embodiment of the present invention;
fig. 7 shows a schematic structural diagram of a data transmission system according to a fourth embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.
Example one
Fig. 2 is a flowchart illustrating a data transmission method according to an embodiment of the present invention. The data transmission method provided by the embodiment of the invention is applied to data transmission from a plurality of upstream devices with hierarchical relationship to a summary device.
It should be noted that the upstream device in any embodiment of the present application may be a data sending device that needs to send data to the summarizing device, and the summarizing device is a data receiving device that needs to receive all data of the upstream device. In the data processing chain, the data transmission device is often located at the front end or upstream of the aggregation device, and therefore the data transmission device is an upstream device.
As shown in fig. 2, the method comprises the steps of:
step S210, generating a hierarchical relationship configuration file of a plurality of upstream devices.
It should be understood that, in the prior art, each upstream device directly transmits data to the aggregation device, and thus, each upstream device in the prior art does not have a hierarchical relationship in the data transmission process, and in the embodiment of the present invention, the upstream device without a hierarchical relationship in the data transmission process in the prior art is configured in a hierarchical relationship, so that the upstream device has a corresponding hierarchical relationship in the data transmission process.
Specifically, in the embodiment of the present invention, the hierarchical relationship between the upstream devices is recorded in the hierarchical relationship configuration file. The embodiment of the present invention does not limit the specific generation manner of the hierarchical relationship configuration file. For example, the hierarchical relationship profile may be generated using a combination of one or more of the following:
the first generation mode is to provide a configuration entry for a user and generate a hierarchical relationship configuration file according to configuration parameters input by the user through the configuration entry. Specifically, a maintenance or analysis person of the data transmission system can specify the hierarchical relationship of the respective upstream devices through the configuration entry. By adopting the mode, the flexibility of user operation can be improved, and the user experience is improved.
And the second generation mode is to extract the service attributes of each upstream device, determine the hierarchical relationship of the upstream devices according to the range relationship of the service attributes of each upstream device aiming at any service attribute, and then generate a hierarchical relationship configuration file. The service attribute includes a geographic location attribute, a category attribute, and the like. For example, if the geographic location attribute value of the upstream device a is location 1 and the geographic location attribute value of the upstream device F is location 2, where the geographic location range of location 2 includes location 1, the upstream device F is determined as the parent upstream device of the upstream device a. Accordingly, if the class attribute value of the upstream device a is class 3 and the class attribute value of the upstream device F is class 4, where the class 4 includes class 3, the upstream device F is determined as the parent upstream device of the upstream device a.
By adopting the hierarchical relationship configuration file provided by the embodiment of the invention, when the upstream equipment needs to be added, deleted and/or the service information of the upstream equipment needs to be changed, the hierarchical relationship configuration file only needs to be changed, the whole data transmission system does not need to be modified, and thus the expandability of the data transmission method and the system is improved.
In addition, in an optional implementation manner, the embodiment of the present invention may further provide a data transfer time configuration entry for a user, where the user may configure data transfer time through the data transfer time configuration entry, and generate a data transfer time configuration file in response to a data transfer time configuration operation of the user. For example, the data transfer time may be 12 hours per day, and so on.
Step S220, for any upstream device, determining whether the upstream device has a parent upstream device according to the hierarchical relationship configuration file, if so, transmitting data corresponding to the upstream device to the parent upstream device of the upstream device, and if not, transmitting data corresponding to the upstream device to the aggregation device.
If a certain upstream device has a parent upstream device, transmitting data corresponding to the certain upstream device to the parent upstream device; and if the certain upstream equipment does not have the parent-level upstream equipment, directly transmitting the data corresponding to the certain upstream equipment to the summarizing equipment.
Specifically, for any upstream device, whether the upstream device has a sublevel upstream device is judged according to a hierarchical relationship configuration file; if so, merging the data uploaded by the sub-level upstream equipment of the upstream equipment and the data stored by the upstream equipment, and taking the merged data as the data corresponding to the upstream equipment; and if not, taking the data stored by the upstream equipment as the data corresponding to the upstream equipment. That is, the upstream device having the child upstream device aggregates the data of each child upstream device of the upstream device and its own data, and then transmits the aggregated data to the parent upstream device or the aggregation device of the upstream device.
In an optional implementation manner, for any upstream device, if the upstream device has a sub-level upstream device, the upstream device determines whether to receive data uploaded by all sub-level upstream devices of the upstream device currently; if yes, merging the data uploaded by all the sub-level upstream devices of the upstream device and the data stored by the upstream device. Therefore, the missing of partial data during data transmission of the upstream equipment can be avoided, and the integrity of data transmission is guaranteed.
In yet another optional implementation, if the user further configures data transmission time, when any upstream device transmits data, the transmission start time is specifically determined according to the data transmission time configuration file, and whether the upstream device has a parent upstream device and subsequent operations is determined according to the hierarchical relationship configuration file at the transmission start time. Thereby enabling timed automatic transmission of data.
In another optional implementation manner, when the upstream device transmits data corresponding to the upstream device, if the data transmission is successful, feedback information of successful data transmission is generated; and if the data transmission fails, generating feedback information of the data transmission failure. If the upstream device has a parent upstream device, the corresponding feedback information is sent to the parent upstream device, and if the upstream device does not have the parent upstream device, the corresponding feedback information is sent to the summarizing device.
Further optionally, the data transmission state of each current upstream device may be monitored in real time, and the data transmission state of each upstream device is displayed in a visual interface; wherein the data transmission state comprises one of the following states: wait for transmission, in data transmission, data transmission success, and data transmission failure. Specifically, if a certain upstream device is in a data transmission process, the data transmission state of the upstream device is in data transmission; if the upstream equipment generates feedback information of successful data transmission, the data transmission state of the upstream equipment is changed into successful data transmission; if the upstream equipment generates feedback information of data transmission failure, changing the data transmission state of the upstream equipment into data transmission failure; if the upstream device has a subordinate upstream device and the upstream device has not received feedback information that the data transmission sent by all subordinate upstream devices of the upstream device is successful, the data transmission state of the upstream device is waiting for transmission. By adopting the embodiment, the node with the error in data transmission can be quickly positioned, and the troubleshooting efficiency is improved.
In addition, in another optional implementation manner, in order to further improve the scalability of the data transmission method and system, a data transmission middleware is generated in the embodiment of the present invention. In the process of transmitting the data corresponding to the upstream device, a data transmission middleware is called through a preset interface, so that the data transmission middleware transmits the data corresponding to the upstream device. Therefore, the embodiment of the invention uniformly transmits data through the data transmission middleware, can avoid the defect of data transmission errors caused by different data transmission apertures among different upstream devices, and improves the accuracy of data transmission. Moreover, if the mode of directly transmitting data by the upstream device is adopted, when the configuration of a certain upstream device is changed, the data transmission interface in the subordinate upstream device, the parent upstream device or the summary device related to the upstream device needs to be adjusted, and by adopting the data transmission middleware in the embodiment mode, only the interface for calling the data transmission middleware needs to be adjusted, so that the expansibility is improved, and the system maintenance cost is reduced.
Further optionally, in the process of transmitting the data corresponding to the upstream device, the data transmission middleware specifically divides a task for transmitting the data corresponding to the upstream device into a plurality of subtasks, and distributes the plurality of subtasks to a plurality of concurrent working threads for execution, thereby improving the data transmission efficiency. In addition, the multiple subtasks can be divided into multiple subtask groups according to the number of the working threads; the number of the subtask groups is consistent with the number of the working threads; and establishing a one-to-one mapping relation between the subtask group and the working thread, and distributing the subtasks in the subtask group to the working thread having the mapping relation with the subtask group for execution, thereby ensuring that the plurality of working threads execute the corresponding subtasks orderly.
Taking fig. 3 as an example, the data transfer middleware takes a data transfer of an upstream device as a task (JOB), and starts a process to complete the task. And reading the data source information and the transmission destination node information of the task through the Cascade Job, and segmenting the task into a plurality of subtasks (Worker, which is the minimum execution unit of the Cascade joba). Specifically, the Capture unit acquires data to be transmitted, and the slice unit performs task segmentation. Then, the Dispatch module is called to determine the number of task groups according to the number of concurrent working threads, and then, a plurality of subtasks are divided into a plurality of subtask groups (groups). Each work thread is responsible for executing subtasks in one subtask group, and the subtask groups and the work threads are in one-to-one correspondence. And each subtask completes the subtask via Read → Channel → Output. And generating feedback information of successful data transmission after the execution of each task group is finished, otherwise generating feedback information of failed data transmission, and sending the feedback information to corresponding parent-level upstream equipment or summarizing equipment.
In step S230, the aggregation device receives data transmitted by an upstream device without a parent upstream device.
Referring to fig. 4, unlike the data transmission method of fig. 1 in the prior art, the embodiment of the present invention hierarchically configures upstream device A, B, C, D, E, F, G, such that upstream device F is a parent upstream device of upstream device A, B, C, and upstream device G is a parent upstream device of upstream device D, E. Accordingly, the upstream device A, B, C transmits its own data to the upstream device F, the upstream device F combines the received data with its own data and transmits the combined data to the aggregation device S, accordingly, the upstream device D, E transmits its own data to the upstream device G, and the upstream device G combines the received data with its own data and transmits the combined data to the aggregation device S. In addition, the data transmission state of each upstream device can be further indicated by different display colors on the basis of fig. 4, so that a user can know the overall data transmission condition conveniently.
Therefore, in the data transmission process from a plurality of upstream devices with hierarchical relationship to a summary device, the embodiment of the present invention generates hierarchical relationship configuration files of the plurality of upstream devices, and determines, for any upstream device, whether the upstream device has a parent upstream device according to the hierarchical relationship configuration files, if so, transmits data corresponding to the upstream device to the parent upstream device of the upstream device, and if not, transmits data corresponding to the upstream device to the summary device; and the summarizing device receives the data transmitted by the upstream device without the upstream device of the parent level. By adopting the scheme, the chaos of the data transmission process can be avoided, and the fault tracing can be rapidly carried out when a certain node breaks down.
Example two
Fig. 5 is a flowchart illustrating a data transmission method according to a second embodiment of the present invention. The execution subject of the embodiment of the invention is any upstream device in a plurality of upstream devices with hierarchical relationship.
As shown in fig. 5, the method includes the steps of:
step S510, judging whether the upstream equipment has parent-level upstream equipment or not according to the generated hierarchical relationship configuration file; if yes, go to step S520; if not, go to step S530.
Step S520, transmitting the data corresponding to the upstream device to the parent upstream device of the upstream device.
Step S530, transmitting the data corresponding to the upstream device to the summarizing device.
In an optional embodiment, the method further comprises: judging whether the upstream equipment has sublevel upstream equipment or not according to the hierarchical relation configuration file;
if so, merging the data transmitted by the subordinate upstream equipment of the upstream equipment and the data stored by the upstream equipment, and taking the merged data as the data corresponding to the upstream equipment;
and if not, taking the data stored in the upstream equipment as the data corresponding to the upstream equipment.
In an optional implementation manner, the transmitting the data corresponding to the upstream device further includes: if the data transmission is successful, generating feedback information of the successful data transmission;
and if the data transmission fails, generating feedback information of the data transmission failure.
In an optional implementation manner, the transmitting the data corresponding to the upstream device further includes:
and calling a data transmission middleware through a preset interface so that the data transmission middleware transmits data corresponding to the upstream equipment.
In an optional embodiment, the method further comprises:
and determining transmission starting time according to the generated data transmission time configuration file, and executing the operation of judging whether the upstream equipment has parent-level upstream equipment according to the generated hierarchical relationship configuration file at the transmission starting time.
The specific implementation process of this embodiment may refer to the description of the corresponding parts in the first embodiment, which is not repeated herein.
Therefore, the embodiment of the invention can avoid the confusion of the data transmission process and quickly trace the source of the fault when a certain node has the fault.
EXAMPLE III
Fig. 6 shows a schematic structural diagram of an upstream device according to a third embodiment of the present invention. As shown in fig. 6, the upstream device 600 includes a determining module 610 and a transmitting module 620.
A judging module 610, configured to judge whether the upstream device has a parent upstream device according to the generated hierarchical relationship configuration file;
a transmission module 620, configured to transmit data corresponding to the upstream device to a parent upstream device of the upstream device if the determination result of the determination module is yes; and if the judgment result of the judgment module is negative, transmitting the data corresponding to the upstream equipment to the summarizing equipment.
In an optional implementation, the determining module 610 is further configured to: judging whether the upstream equipment has sub-level upstream equipment or not according to the hierarchical relation configuration file; the transmission module 620 is further configured to, if the upstream device has a sub-level upstream device, merge data transmitted by the sub-level upstream device of the upstream device with data stored in the upstream device, and use the merged data as data corresponding to the upstream device; and if the upstream equipment does not have the sub-level upstream equipment, taking the data stored by the upstream equipment as the data corresponding to the upstream equipment.
In an optional implementation manner, the upstream device 600 further includes a generating module (not shown in the figure) configured to generate feedback information that the data transmission is successful if the data transmission is successful; and if the data transmission fails, generating feedback information of the data transmission failure.
In an alternative embodiment, the transmission module 620 is further configured to: and calling the data transmission middleware through a preset interface so that the data transmission middleware can transmit the data corresponding to the upstream equipment.
In an optional implementation, the determining module 610 is further configured to: and determining transmission starting time according to the generated data transmission time configuration file, and executing the operation of judging whether the upstream equipment has parent-level upstream equipment according to the generated hierarchical relationship configuration file at the transmission starting time.
The specific implementation process of each module in this embodiment may refer to the description in the method embodiment, and is not described herein again.
Therefore, the embodiment of the invention can avoid the confusion of the data transmission process and quickly trace the source of the fault when a certain node has the fault.
Example four
Fig. 7 shows a schematic structural diagram of a data transmission system according to a fourth embodiment of the present invention. As shown in fig. 7, the data transmission system 700 includes: a plurality of upstream devices 600, a summarizing device 710, and a configuration module 720.
The specific structure of the upstream device 600 can refer to the corresponding description in the third embodiment, and is not described herein again.
And the summarizing device 710 is used for receiving data transmitted by the upstream device without the parent-level upstream device.
A configuration module 720, configured to generate a hierarchical relationship configuration file of the plurality of upstream devices.
In an alternative embodiment, the data transmission system 700 further includes a display module (not shown in the figure) for displaying the data transmission status of each upstream device in the visual interface; wherein the data transmission state comprises one of the following states: wait for transmission, in data transmission, data transmission success, and data transmission failure.
In an alternative embodiment, the data transmission system 700 further includes data transmission middleware (not shown) for transmitting data corresponding to the upstream device.
In an alternative embodiment, the data transfer middleware is configured to: and splitting a task for transmitting data corresponding to the upstream equipment into a plurality of subtasks, and distributing the subtasks to a plurality of concurrent working threads for execution.
In an alternative embodiment, the data transfer middleware is configured to: dividing the plurality of subtasks into a plurality of subtask groups according to the number of the working threads; wherein the number of the subtask groups is consistent with the number of the working threads; and establishing a one-to-one mapping relation between the subtask group and the working thread, and distributing the subtasks in the subtask group to the working thread having the mapping relation with the subtask group for execution.
In an alternative embodiment, the configuration module 720 is further configured to generate a data transfer time configuration file in response to a data transfer time configuration operation of a user.
The specific implementation process of the system may refer to the description in the corresponding method embodiment, which is not described herein again.
Therefore, the embodiment of the invention can avoid the confusion of the data transmission process and quickly trace the source of the fault when a certain node has the fault.
EXAMPLE five
An embodiment five of the present invention provides a non-volatile computer storage medium, where the computer storage medium stores at least one executable instruction, and the computer executable instruction may execute the method in any of the above method embodiments.
The executable instructions may be specifically configured to cause the processor to:
judging whether the upstream equipment has parent-level upstream equipment or not according to the generated hierarchical relationship configuration file;
if so, transmitting the data corresponding to the upstream equipment to the parent upstream equipment of the upstream equipment;
and if not, transmitting the data corresponding to the upstream equipment to the summarizing equipment.
In an alternative embodiment, the executable instructions may be specifically configured to cause the processor to:
judging whether the upstream equipment has sublevel upstream equipment or not according to the hierarchical relation configuration file;
if so, merging the data transmitted by the subordinate upstream equipment of the upstream equipment and the data stored by the upstream equipment, and taking the merged data as the data corresponding to the upstream equipment;
and if not, taking the data stored in the upstream equipment as the data corresponding to the upstream equipment.
In an alternative embodiment, the executable instructions may be specifically configured to cause the processor to:
if the data transmission is successful, generating feedback information of the successful data transmission;
and if the data transmission fails, generating feedback information of the data transmission failure.
In an alternative embodiment, the executable instructions may be specifically configured to cause the processor to:
and calling a data transmission middleware through a preset interface so that the data transmission middleware transmits data corresponding to the upstream equipment.
In an alternative embodiment, the executable instructions may be specifically configured to cause the processor to:
and determining transmission starting time according to the generated data transmission time configuration file, and executing the operation of judging whether the upstream equipment has parent-level upstream equipment according to the generated hierarchical relationship configuration file at the transmission starting time.
In an alternative embodiment, the executable instructions may be specifically configured to cause the processor to:
generating a hierarchical relationship configuration file for the plurality of upstream devices.
In an alternative embodiment, the executable instructions may be specifically configured to cause the processor to:
displaying the data transmission state of each upstream device in a visual interface;
wherein the data transmission state comprises one of the following states: wait for transmission, in data transmission, data transmission success, and data transmission failure.
In an alternative embodiment, the executable instructions may be specifically configured to cause the processor to:
and splitting a task for transmitting data corresponding to the upstream equipment into a plurality of subtasks, and distributing the subtasks to a plurality of concurrent working threads for execution.
In an alternative embodiment, the executable instructions may be specifically configured to cause the processor to:
dividing the plurality of subtasks into a plurality of subtask groups according to the number of the working threads; wherein the number of the subtask groups is consistent with the number of the working threads;
and establishing a one-to-one mapping relation between the subtask group and the working thread, and distributing the subtasks in the subtask group to the working thread having the mapping relation with the subtask group for execution.
In an alternative embodiment, the executable instructions may be specifically configured to cause the processor to:
and generating a data transmission time configuration file in response to the data transmission time configuration operation of the user.
The algorithms or displays presented herein are not inherently related to any particular computer, virtual system, or other apparatus. Various general purpose systems may also be used with the teachings herein. The required structure for constructing such a system will be apparent from the description above. In addition, embodiments of the present invention are not directed to any particular programming language. It is appreciated that a variety of programming languages may be used to implement the teachings of the present invention as described herein, and any descriptions of specific languages are provided above to disclose the best mode of the invention.
In the description provided herein, numerous specific details are set forth. It is understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.
Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the embodiments of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the invention and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be interpreted as reflecting an intention that: that the invention as claimed requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.
Those skilled in the art will appreciate that the modules in the device in an embodiment may be adaptively changed and disposed in one or more devices different from the embodiment. The modules or units or components of the embodiments may be combined into one module or unit or component, and furthermore they may be divided into a plurality of sub-modules or sub-units or sub-components. All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or elements of any method or apparatus so disclosed, may be combined in any combination, except combinations where at least some of such features and/or processes or elements are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.
Furthermore, those skilled in the art will appreciate that while some embodiments herein include some features included in other embodiments, rather than other features, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.
The various component embodiments of the invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that a microprocessor or Digital Signal Processor (DSP) may be used in practice to implement some or all of the functionality of some or all of the components according to embodiments of the present invention. The present invention may also be embodied as apparatus or device programs (e.g., computer programs and computer program products) for performing a portion or all of the methods described herein. Such programs implementing the present invention may be stored on computer-readable media or may be in the form of one or more signals. Such a signal may be downloaded from an internet website or provided on a carrier signal or in any other form.
It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The usage of the words first, second and third, etcetera do not indicate any ordering. These words may be interpreted as names. The steps in the above embodiments should not be construed as limiting the order of execution unless specified otherwise.
Claims (16)
1. A data transmission method is applied to data transmission from a plurality of upstream devices with hierarchical relationship to a summary device, and comprises the following steps:
generating a hierarchical relationship configuration file of the plurality of upstream devices;
for any upstream device, judging whether the upstream device has a parent upstream device or not according to the hierarchical relationship configuration file, if so, transmitting data corresponding to the upstream device to the parent upstream device of the upstream device, and if not, transmitting data corresponding to the upstream device to the summarizing device;
the aggregation equipment receives data transmitted by upstream equipment without parent upstream equipment.
2. The method of claim 1, further comprising:
for any upstream equipment, judging whether the upstream equipment has sub-level upstream equipment or not according to the hierarchical relation configuration file;
if so, merging the data transmitted by the sub-level upstream equipment of the upstream equipment and the data stored by the upstream equipment, and taking the merged data as the data corresponding to the upstream equipment;
and if not, taking the data stored by the upstream equipment as the data corresponding to the upstream equipment.
3. The method of claim 1, wherein transmitting the data corresponding to the upstream device further comprises: if the data transmission is successful, generating feedback information of the successful data transmission;
and if the data transmission fails, generating feedback information of the data transmission failure.
4. The method of claim 3, further comprising:
displaying the data transmission state of each upstream device in a visual interface;
wherein the data transmission state comprises one of the following states: wait for transmission, in data transmission, data transmission success, and data transmission failure.
5. The method of any of claims 1-4, wherein transmitting the data corresponding to the upstream device further comprises:
and calling a data transmission middleware through a preset interface so that the data transmission middleware can transmit data corresponding to the upstream equipment.
6. The method of claim 5, wherein the data transmission middleware transmitting the data corresponding to the upstream device further comprises:
and the data transmission middleware divides a task for transmitting data corresponding to the upstream equipment into a plurality of subtasks and distributes the subtasks to a plurality of concurrent working threads for execution.
7. The method of claim 6, wherein distributing the plurality of subtasks to a plurality of concurrent worker thread executions further comprises:
dividing the plurality of subtasks into a plurality of subtask groups according to the number of the working threads; wherein the number of the subtask groups is consistent with the number of the working threads;
and establishing a one-to-one mapping relation between the subtask group and the working thread, and distributing the subtasks in the subtask group to the working thread having the mapping relation with the subtask group for execution.
8. The method according to any one of claims 1-4, further comprising:
responding to the data transmission time configuration operation of a user, and generating a data transmission time configuration file;
the determining, for any upstream device, whether the upstream device has a parent upstream device according to the hierarchical relationship configuration file further includes: and any upstream equipment determines transmission starting time according to the data transmission time configuration file, and executes the operation of judging whether the upstream equipment has parent-level upstream equipment or not according to the hierarchical relationship configuration file at the transmission starting time.
9. A data transmission method, characterized in that the method is executed by any one of a plurality of upstream apparatuses having a hierarchical relationship; the method comprises the following steps:
judging whether the upstream equipment has parent-level upstream equipment or not according to the generated hierarchical relationship configuration file;
if so, transmitting the data corresponding to the upstream equipment to the parent upstream equipment of the upstream equipment;
and if not, transmitting the data corresponding to the upstream equipment to the summarizing equipment.
10. The method of claim 9, further comprising:
judging whether the upstream equipment has sublevel upstream equipment or not according to the hierarchical relation configuration file;
if so, merging the data transmitted by the subordinate upstream equipment of the upstream equipment and the data stored by the upstream equipment, and taking the merged data as the data corresponding to the upstream equipment;
and if not, taking the data stored in the upstream equipment as the data corresponding to the upstream equipment.
11. The method of claim 9, wherein the transmitting the data corresponding to the upstream device further comprises: if the data transmission is successful, generating feedback information of the successful data transmission;
and if the data transmission fails, generating feedback information of the data transmission failure.
12. The method according to any one of claims 9-11, wherein the transmitting the data corresponding to the upstream device further comprises:
and calling a data transmission middleware through a preset interface so that the data transmission middleware transmits data corresponding to the upstream equipment.
13. The method according to any one of claims 9-11, further comprising:
and determining transmission starting time according to the generated data transmission time configuration file, and executing the operation of judging whether the upstream equipment has parent-level upstream equipment according to the generated hierarchical relationship configuration file at the transmission starting time.
14. An upstream device, comprising:
the judging module is used for judging whether the upstream equipment has parent-level upstream equipment or not according to the generated hierarchical relationship configuration file;
the transmission module is used for transmitting the data corresponding to the upstream equipment to the parent upstream equipment of the upstream equipment if the judgment result of the judgment module is positive; and if the judgment result of the judgment module is negative, transmitting the data corresponding to the upstream equipment to the summarizing equipment.
15. A data transmission system, comprising: a plurality of upstream devices, aggregation devices, and configuration modules according to claim 14;
the configuration module is used for generating a hierarchical relationship configuration file of the plurality of upstream devices;
the summarizing device is used for receiving data transmitted by the upstream device without the parent-level upstream device.
16. A computer storage medium having at least one executable instruction stored therein, the executable instruction causing a processor to perform operations corresponding to the data transmission method according to any one of claims 9-13.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111631053.XA CN114363411B (en) | 2021-12-28 | 2021-12-28 | Data transmission method, device, system and computer storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111631053.XA CN114363411B (en) | 2021-12-28 | 2021-12-28 | Data transmission method, device, system and computer storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114363411A true CN114363411A (en) | 2022-04-15 |
| CN114363411B CN114363411B (en) | 2023-11-17 |
Family
ID=81102813
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111631053.XA Active CN114363411B (en) | 2021-12-28 | 2021-12-28 | Data transmission method, device, system and computer storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114363411B (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020099690A1 (en) * | 2001-01-16 | 2002-07-25 | Schumacher Michael K. | System and method for managing information for a plurality of computer systems in a distributed network |
| US20110078691A1 (en) * | 2009-09-30 | 2011-03-31 | Microsoft Corporation | Structured task hierarchy for a parallel runtime |
| CN104765866A (en) * | 2015-04-24 | 2015-07-08 | 武汉噢易云计算有限公司 | Method and system for administrative division data gathering based on tree structure |
| US20170366513A1 (en) * | 2016-06-17 | 2017-12-21 | Cisco Technology, Inc. | Data Anonymization for Distributed Hierarchical Networks |
| US9906413B1 (en) * | 2014-12-18 | 2018-02-27 | Jpmorgan Chase Bank, N.A. | System and method for implementing a dynamic hierarchy for devices |
| WO2020103736A1 (en) * | 2018-11-23 | 2020-05-28 | 阿里巴巴集团控股有限公司 | Data transmission device, processing system, and message distribution method and apparatus |
| CN111813554A (en) * | 2020-07-17 | 2020-10-23 | 济南浪潮数据技术有限公司 | Task scheduling processing method and device, electronic equipment and storage medium |
| CN112217657A (en) * | 2019-07-11 | 2021-01-12 | 奇安信科技集团股份有限公司 | Data transmission method, data processing method, device and medium based on SD-WAN system |
| CN112597210A (en) * | 2020-12-16 | 2021-04-02 | 中国建设银行股份有限公司 | Data summarizing method, device, equipment and storage medium |
-
2021
- 2021-12-28 CN CN202111631053.XA patent/CN114363411B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020099690A1 (en) * | 2001-01-16 | 2002-07-25 | Schumacher Michael K. | System and method for managing information for a plurality of computer systems in a distributed network |
| US20110078691A1 (en) * | 2009-09-30 | 2011-03-31 | Microsoft Corporation | Structured task hierarchy for a parallel runtime |
| US9906413B1 (en) * | 2014-12-18 | 2018-02-27 | Jpmorgan Chase Bank, N.A. | System and method for implementing a dynamic hierarchy for devices |
| CN104765866A (en) * | 2015-04-24 | 2015-07-08 | 武汉噢易云计算有限公司 | Method and system for administrative division data gathering based on tree structure |
| US20170366513A1 (en) * | 2016-06-17 | 2017-12-21 | Cisco Technology, Inc. | Data Anonymization for Distributed Hierarchical Networks |
| WO2020103736A1 (en) * | 2018-11-23 | 2020-05-28 | 阿里巴巴集团控股有限公司 | Data transmission device, processing system, and message distribution method and apparatus |
| CN112217657A (en) * | 2019-07-11 | 2021-01-12 | 奇安信科技集团股份有限公司 | Data transmission method, data processing method, device and medium based on SD-WAN system |
| CN111813554A (en) * | 2020-07-17 | 2020-10-23 | 济南浪潮数据技术有限公司 | Task scheduling processing method and device, electronic equipment and storage medium |
| CN112597210A (en) * | 2020-12-16 | 2021-04-02 | 中国建设银行股份有限公司 | Data summarizing method, device, equipment and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 查选: "物联网数据安全传输相关问题研究", 《中国博士学位论文全文数据库 (信息科技辑)》, pages 136 - 125 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114363411B (en) | 2023-11-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109313739B (en) | System and method for providing visualization of workflow | |
| US10909028B1 (en) | Multi-version regression tester for source code | |
| CN105373428B (en) | Task scheduling method and system | |
| US20150100832A1 (en) | Method and system for selecting and executing test scripts | |
| US20150100829A1 (en) | Method and system for selecting and executing test scripts | |
| US20150100830A1 (en) | Method and system for selecting and executing test scripts | |
| CN108763009B (en) | Server stress test method, system, equipment and computer readable storage medium | |
| US20150100831A1 (en) | Method and system for selecting and executing test scripts | |
| CN111831191A (en) | Workflow configuration method and device, computer equipment and storage medium | |
| CN112799782B (en) | Model generation system, method, electronic device and storage medium | |
| CN110673842A (en) | Visual programming method and system, storage medium and equipment | |
| CN109905263A (en) | A kind of automation O&M deployment system | |
| CN111459631A (en) | Automatic batch processing method and system for server | |
| CN110232013B (en) | Test method, test device, controller and medium | |
| CN113658351A (en) | Product production method and device, electronic equipment and storage medium | |
| CN114363411B (en) | Data transmission method, device, system and computer storage medium | |
| CN107025126B (en) | Resource scheduling method, NFVO and system | |
| US20230061780A1 (en) | Automated end-to-end testing platform with dynamic container configuration | |
| CN111966597B (en) | Test data generation method and device | |
| CN113419829B (en) | Job scheduling method, device, scheduling platform and storage medium | |
| CN111897725B (en) | Automatic test method, medium, equipment and system for middle platform service | |
| CN114004477A (en) | Low-code-based automatic execution method and device for process arrangement | |
| CN111581042B (en) | Cluster deployment method, deployment platform and server to be deployed | |
| CN112183982A (en) | Workflow creating method and device, computer equipment and storage medium | |
| CN110837399A (en) | Method and device for managing streaming computing application program and computing equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: Room 332, 3 / F, Building 102, 28 xinjiekouwei street, Xicheng District, Beijing 100088 Applicant after: Qianxin Technology Group Co.,Ltd. Applicant after: Qianxin Wangshen information technology (Beijing) Co.,Ltd. Address before: Room 332, 3 / F, Building 102, 28 xinjiekouwei street, Xicheng District, Beijing 100088 Applicant before: Qianxin Technology Group Co.,Ltd. Applicant before: LEGENDSEC INFORMATION TECHNOLOGY (BEIJING) Inc. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |