CN117880276A - Transmission task scheduling method and application based on nginx - Google Patents

Transmission task scheduling method and application based on nginx Download PDF

Info

Publication number
CN117880276A
CN117880276A CN202410064418.2A CN202410064418A CN117880276A CN 117880276 A CN117880276 A CN 117880276A CN 202410064418 A CN202410064418 A CN 202410064418A CN 117880276 A CN117880276 A CN 117880276A
Authority
CN
China
Prior art keywords
oversized
agent
file transmission
transmission
oversized file
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202410064418.2A
Other languages
Chinese (zh)
Inventor
廖非凡
严其飞
米海波
刘凤增
周书林
徐振兴
王�锋
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
National University of Defense Technology
Original Assignee
National University of Defense Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by National University of Defense Technology filed Critical National University of Defense Technology
Priority to CN202410064418.2A priority Critical patent/CN117880276A/en
Publication of CN117880276A publication Critical patent/CN117880276A/en
Pending legal-status Critical Current

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14—Error detection or correction of the data by redundancy in operation
    • G06F11/1402—Saving, restoring, recovering or retrying
    • G06F11/1446—Point-in-time backing up or restoration of persistent data
    • G06F11/1458—Management of the backup or restore process
    • G06F11/1469—Backup restoration techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/18—File system types
    • G06F16/185—Hierarchical storage management [HSM] systems, e.g. file migration or policies thereof
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/14—Session management
    • H04L67/141—Setup of application sessions
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/56—Provisioning of proxy services

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Quality & Reliability (AREA)
  • Computer And Data Communications (AREA)

Abstract

The invention discloses a transmission task scheduling method based on nginx, which comprises the following steps: creating an oversized file transmission agent cluster based on the oversized file transmission agent, and establishing communication connection with each oversized file transmission agent in the oversized file transmission agent cluster by utilizing a reverse agent mechanism of the nginx service; sensing state information of the oversized file transmission agent and taking the state information as a decision basis of a task scheduling strategy, and combining message characteristics to obtain a decision result; transmitting the message to an oversized file transmission agent according to the decision result, wherein the oversized file transmission agent is responsible for forwarding the message and writing the corresponding message information into a shared database table; and sensing whether the oversized file transmission agent fails or not in real time, and if so, enabling the failed node to durably support the file migration recovery algorithm. The method can solve the problems that the existing oversized file transmission agent can only operate by a single node, and communication data volume cannot meet the increasingly high concurrency, high data volume and other communication problems due to the upper limit of the performance of hardware equipment.

Description

Transmission task scheduling method and application based on nginx
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a transmission task scheduling method based on nginx, a transmission task scheduling system based on nginx, an electronic device, and a computer readable storage medium.
Background
The ultra-large file transmission service realizes the service of the cross-domain reliable transmission of the message by providing a transmission interface conforming to the standard definition for an upper user. The ultra-large file transmission service software has asynchronous transmission characteristics, and supports that two communication ends are not required to be in an operation state; the method has the breakpoint continuous transmission characteristic, and can directly resume transmission from the interruption of the transmission path after the failure of receiving or transmitting the message. The ultra-large file transmission service consists of an ultra-large file transmission agent and an ultra-large file transmission application support library, provides reliable transmission capacity facing a wide area network for file and message transmission of various application systems, supports a domestic CPU (Central processing Unit) computer and supports running in a domestic operating system and a database.
With the gradual development of the current network environment, especially the upgrade of network hardware equipment and facilities, the requirement of the whole application system on the communication data volume is also increased. In the above background environment, because the communication data volume that can be processed by the oversized file transfer agent operated by a single node has an upper limit due to the performance of the hardware device itself, the ever-increasing communication data volume transmission requirement is no longer satisfied.
Disclosure of Invention
Aiming at the defects or improvement demands of the prior art, the invention provides a transmission task scheduling method and application based on nginx, which can solve the problems that the existing oversized file transmission agent can only operate by a single node, and the communication data volume cannot meet the increasingly high concurrency, high data volume and other communication problems due to the upper limit of the performance of hardware equipment.
In one aspect, an embodiment of the present invention provides a transmission task scheduling method based on nginx, including: creating an oversized file transmission agent cluster based on the oversized file transmission agent, and establishing communication connection with each oversized file transmission agent in the oversized file transmission agent cluster by utilizing a reverse agent mechanism of the nginx service; sensing state information of an oversized file transmission agent in the oversized file transmission agent cluster, taking the state information as a decision basis of a transmission task scheduling strategy, and combining message characteristics to obtain a decision result; transmitting the message to a corresponding oversized file transmission agent in the oversized file transmission agent cluster according to the decision result, wherein the oversized file transmission agent is responsible for forwarding the message and writing the corresponding message information into a shared database table; whether the oversized file transmission agent in the oversized file transmission agent cluster breaks down or not is perceived in real time, and if so, the fault node persists the file migration recovery algorithm, so that the normal operation of the oversized file transmission agent cluster is ensured.
In one embodiment of the present invention, the establishing a communication connection with each oversized file transfer agent within the oversized file transfer agent cluster includes: the method comprises the steps that a link establishment request is sent to an external IP address and a port of an oversized file transmission proxy cluster, and the link establishment request is waited to be fed back, if waiting time is longer than a preset time length, the link establishment fails; otherwise, acquiring IP of all oversized file transfer agents in the oversized file transfer agent cluster i Address and port set, traversing each IP in the port set i Address and try IP i The address establishes a remote communication link, if the link still fails to be established, the IP is used for the link establishment i Adding the address into a retry address set; when the retry address set is a non-empty set, initiating a link establishment confirmation, and confirmingIP of (c) i Adding the address into the port set again, and emptying the retry address set; repeating the above process until all the IPs i The address successfully establishes the communication link.
In one embodiment of the present invention, the transmission task scheduling method based on nginx further includes: and after the ultra-large file transmission proxy cluster is updated, updating the communication connection based on the new ultra-large file transmission proxy cluster when a new message transmission task is initiated.
In one embodiment of the invention, the status information includes: static information representing the joining state of the oversized file transfer agent when the oversized file transfer agent is added into the oversized file transfer agent cluster; and dynamic information characterizing the operational state of the oversized file transfer agent when running.
In one embodiment of the present invention, the decision result obtaining process includes: presetting reference weight values of different operation parameters n of the oversized file transfer agent and using a set P i The representation, wherein,generating a set of motion state descriptions based on dynamic information +.>Wherein,a numerical value representing different operating parameters n of the oversized file transfer agent; acquiring a message x running state weight adjustment value set P x ，/>Wherein (1)>Representing the message x for different operating parameters x n Is a set of the adjustment values of (2); when a message x is sent, calculating the running state comprehensive weight of the oversized file transmission agent as follows: comprehensive weight based on running state>The size of the super-large file transmission agent of the message x is randomly determined, and the running state comprehensive weight of the super-large file transmission agent is +.>Probability PR of becoming the very large file transfer agent i The calculated relation of (2) is:
in one embodiment of the present invention, the failed node persistent file migration recovery algorithm includes: marking the failed oversized file transfer agent as a failed oversized file transfer agent, wherein the failed oversized file transfer agent is no longer a target of a transfer task; initiating a persistent file migration instruction to a daemon on a fault oversized file transfer agent node, transferring the file which is completely transferred to the fault oversized file transfer agent node to other oversized file transfer agent nodes, and taking charge of forwarding the message by other oversized file transfer agent nodes.
In one embodiment of the present invention, the fault determination procedure includes: when the oversized file transmission agent joins the oversized file transmission agent cluster, reporting self information and periodically reporting the survivability of the oversized file transmission agent through a heartbeat mechanism; and sending a heartbeat message by the oversized file transmission agent every preset heartbeat period, and if the heartbeat message from the oversized file transmission agent is not received in a preset heartbeat check period, regarding the oversized file transmission agent as a heartbeat timeout, and judging that the oversized file transmission agent is a transmission service fault.
On the other hand, the embodiment of the invention provides a transmission task scheduling system based on the nginx, which comprises the following steps: the transmission link establishment module is used for establishing communication connection with the oversized file transmission agent in the oversized file transmission agent cluster by utilizing a reverse agent mechanism of the nginx service; the task scheduling and forwarding module is used for transmitting the message to a corresponding oversized file transmission agent in the oversized file transmission agent cluster based on a transmission task scheduling strategy, and the oversized file transmission agent is responsible for forwarding the message and writing corresponding message information into a shared database table; the fault recovery module is arranged to enable the whole oversized file transmission agent cluster to normally operate when the oversized file transmission agent in the oversized file transmission agent cluster breaks down, and support the restoration of the broken oversized file transmission agent and then rejoin the cluster.
In still another aspect, an embodiment of the present invention provides an electronic device, including: a memory and one or more processors coupled to the memory, the memory storing a computer program for executing the computer program to perform the steps of the method according to any of the embodiments described above.
In yet another aspect, embodiments of the present invention provide a computer-readable storage medium storing computer-executable instructions for performing the steps of the method according to any one of the embodiments described above.
From the above, the above scheme contemplated by the present invention may have the following compared with the prior art
The beneficial effects are that:
by clustering the oversized file transmission agents, the mode that a plurality of oversized file transmission agents share the same external address is realized, the oversized file transmission agents jointly process data communication which can only be forwarded by one oversized file transmission agent, when a system needs higher content data traffic, the oversized file transmission agent cluster is only required to be expanded, and the problem that the internal data traffic of the system is gradually increased, and the oversized file transmission agents deployed by a single node cannot meet the increasingly-growing transmission traffic demand due to the fact that the capacity of hardware equipment per se is limited. In addition, the fault tolerance and fault recovery functions are expanded on the basis of the transmission agent clusters, and the robustness of the transmission scheduling system is enhanced.
Other aspects of the features of the invention will become apparent from the following detailed description, which refers to the accompanying drawings. It is to be understood, however, that the drawings are designed solely for the purposes of illustration and not as a definition of the limits of the invention. It should be further understood that the drawings are not necessarily drawn to scale and that, unless otherwise indicated, they are merely intended to conceptually illustrate the structures and procedures described herein.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the invention and do not constitute a limitation on the invention. In the drawings:
fig. 1 is a flowchart of a transmission task scheduling method based on nginx according to an embodiment of the present invention;
FIG. 2 is a schematic topology diagram of an oversized file transfer agent cluster according to an embodiment of the present invention;
fig. 3 is a schematic diagram of specific execution steps of a transmission task scheduling method based on nginx according to an embodiment of the present invention;
fig. 4 is a schematic diagram of a scheduling service architecture of a transmission task scheduling method based on nginx according to an embodiment of the present invention;
fig. 5 is a transmission link establishment process diagram of a transmission task scheduling method based on nginx according to an embodiment of the present invention;
fig. 6 is a communication task forwarding process diagram of a transmission task scheduling method based on nginx according to an embodiment of the present invention;
fig. 7 is a fault recovery process diagram of a transmission task scheduling method based on nginx according to an embodiment of the present invention;
fig. 8 is a schematic structural diagram of a transmission task scheduling system based on nginx according to an embodiment of the present invention;
fig. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present invention;
fig. 10 is a schematic structural diagram of a computer readable storage medium according to an embodiment of the present invention.
Detailed Description
It should be noted that, without conflict, the embodiments of the present invention and features of the embodiments may be combined with each other. The invention will be described below with reference to the accompanying drawings in combination with embodiments.
In order to enable those skilled in the art to better understand the technical solutions of the present invention, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are only some embodiments of the present invention, but not all embodiments of the embodiments are all within the protection scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the above figures are applicable to distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the terms so used are interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or otherwise described herein. Furthermore, it is possible to provide a device for the treatment of a disease. The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed or inherent to such process, method, article, or apparatus, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
It should be further noted that the division of the embodiments in the present invention is only for convenience of description, and should not be construed as a specific limitation, and features in the various embodiments may be combined and mutually referenced without contradiction.
As shown in fig. 1, a first embodiment of the present invention proposes a transmission task scheduling method based on nginx, for example, including: step S1, an oversized file transmission agent cluster is established based on an oversized file transmission agent, and communication connection is established between the oversized file transmission agent cluster and each oversized file transmission agent in the oversized file transmission agent cluster by using a reverse agent mechanism of the nginx service; step S2, sensing the state information of the oversized file transmission agents in the oversized file transmission agent cluster, taking the state information as a decision basis of a transmission task scheduling strategy, and combining message characteristics to obtain a decision result; step S3, transmitting the message to a corresponding oversized file transmission agent in the oversized file transmission agent cluster according to the decision result, wherein the oversized file transmission agent is responsible for forwarding the message and writing the corresponding message information into a shared database table; and S4, sensing whether the oversized file transfer agent in the oversized file transfer agent cluster is faulty in real time, and if so, enabling the faulty node to persist the file transfer recovery algorithm to ensure the normal operation of the oversized file transfer agent cluster.
Specifically, the basic concept used in the present invention will be described with reference to fig. 2:
(1) Oversized file transfer agent: each sub-network is not at least an application program of a node and is used for forwarding a message transmitted by the local area network to the wide area network or a message transmitted by the wide area network to the local area network.
(2) Oversized file transfer agent cluster: an application cluster consisting of a dispatch service, 1 and more oversized file transfer agents is logically considered as one oversized file transfer agent (all oversized file transfer agents within the cluster share the same IP address and port).
(3) Scheduling services: the control end of the oversized file transmission agent cluster is used for managing the states of all oversized file transmission agents and transmission task scheduling in the whole cluster.
(4) Global information view: the information set of all oversized file transmission agents in the corresponding cluster maintained by the scheduling service is mainly used for transmission chain establishment and transmission task scheduling.
The steps are described below in conjunction with the flow shown in fig. 3: in step S1, an oversized file transfer agent is basedCreating a very large file transfer proxy cluster A Largefile Wherein, the method comprises the steps of, wherein,i is an integer greater than 1.
A transmission link establishment stage: the remote transmission application establishes communication connection with each oversized file transmission agent in the oversized file transmission agent cluster through a reverse agent mechanism of the nginx service, and the establishment flow of the communication connection is as follows:
generating a link establishment request to an external IP address and a port of the ultra-large file transmission proxy cluster-ginx service, and waiting for feedback of the link establishment request: if the waiting time length is longer than the preset time length, the link establishment fails; otherwise, acquiring the IP of all the oversized file transfer agents in the oversized file transfer agent cluster i Address and port set address vec, then traverse each IP in the set i Address and try IP i Address establishes a remote communication link, if the link establishment fails or times out, IP is established i Adding the address into a retry address set address Vec_retry;
if the retry address Vec_retry set is a non-empty set, initiating a chain establishment acknowledgment and validating the IP i Adding the address into the address Vec set again, and clearing the retry address Vec_retry set; and repeating the steps until all the addresses successfully establish the communication link, and establishing the link successfully.
In other words, in the oversized file transmission proxy cluster, one node needs to run a scheduling service, where the scheduling service includes an nginx service, and exposes an external IP address and a port of the nginx service as an external IP address and a port of the whole oversized file transmission proxy cluster to a remote communication application. When the remote communication application accesses the external IP address and port of the nginx service, the address of each transmission agent in the oversized file transmission agent cluster is acquired, and connection is correspondingly established.
A detailed description of the transmission chaining algorithm is shown in algorithm 1.
Algorithm 1 transmission link establishment algorithm input: the ultra-large file transmission proxy cluster outputs an external IP address and a port: chain building fruit
When oversized file transmission agent cluster A Largefile After the update, when a new message transmission task is initiated, the new ultra-large file transmission agent cluster A is based Largefile Updating the communication connection: and after the transmission task is overtime, the remote communication task which is initiating the message transmission task to the downward transmission agent carries out transmission link establishment again by revising the external IP address and the port of the nginx service.
A detailed description of the retransmission chain is shown in algorithm 2.
Algorithm 2 retransmission chain-building algorithm
Input: perceiving link disconnection (heartbeat timeout)
And (3) outputting: rebuilding chain results
In step S2, the communication task schedules the forwarding phase: perceived ultra-large file transfer proxy cluster A Largefile In an oversized file transfer agentAnd taking the state information as a decision basis of a task scheduling strategy, and combining message characteristics to obtain a decision result.
In step S3, the message is transmitted to the ultra-large file transmission agent cluster A according to the decision result Largefile Internal oversized file transfer agentWherein j is more than or equal to 1 and less than or equal to i; said extra large file transfer agent->And obtaining a decision result, and taking charge of forwarding the message and writing the corresponding message information into a common database table.
Wherein it should be ensured that the scheduling service is able to perceive the status information of each agent in the oversized file transfer agent cluster as the basis of the transmission task scheduling policy, so in this embodiment the status information comprises information about the oversized file transfer agentOn-line status of (c) and corresponding status information.
Wherein the online status includes: oversized file transfer agentAdding an oversized file transfer agent cluster A Large file The corresponding state information is static information of the super Large file transmission agent Large file, and the static information is used for describing the characteristics and the capabilities of the super Large file transmission agent Large file. Oversized file transfer agent->The running state in running and the corresponding state information is super large file transmission agent +.>Dynamic information describing its operating state.
Illustrating: when the remote communication application initiates a transmission task, a transmission task request is initiated to a scheduling service, a transmission destination address based on a scheduling policy is acquired from the scheduling service, and the scheduling policy should be based on different message characteristics and actual running states of a transmission agent cluster to decide the transmission scheduling policy, for example:
(1) Messages with smaller message length, higher real-time requirement and higher priority (for example, command data type messages) should be forwarded by an oversized file transfer agent with higher performance (high CPU utilization rate) in priority;
(2) The message length is longer, the real-time requirement is not high, the message (for example, situation data type message) with low priority should be forwarded by the oversized file transfer agent with larger disk space and more idle priority.
Because the running states of the oversized file transmission agents have various parameter descriptions, the requirements of different characteristic messages on different running parameters are different, different running parameter weights are preset according to message characteristics, and thus a final comprehensive weight value is calculated, and the scheduling transmission destination agent is selected.
Further, the decision result obtaining flow is as follows:
presetting about oversized file transfer agentReference weight values of different operating parameters n of (a) and using a set P i Representation, wherein->
Generating a set of motion state descriptions based on dynamic informationWherein (1)>Representing an oversized file transfer agent->Specific values of the different operating parameters n. E.g. bandwidth rate usage S 0 47% CPU utilization S 1 25%.
Acquiring a message x running state weight adjustment value set P x ，Wherein,representing the message x for different operating parameters x n Is a set of the adjustment values of (2);
when the remote communication application transmits the message x, the following formula is adopted to calculate the oversized file transmission agentIs added to the running state comprehensive weight->
Comprehensive weight V based on running state xi Is used for randomly determining the size of the super-large file transmission agent of the message xWherein, the extra large file transfer agent->Is added to the running state comprehensive weight->The larger it becomes the oversized file transfer agent +.>The higher the probability of (i.e. oversized file transfer agent +.>) Defined as the destination of the message, the probability PR i The calculation mode of (2) is as follows:
in step S4, during the fault recovery phase, when the real-time sensing is performedUltra-large file transfer proxy cluster A Large file Internal oversized file transfer agentWhether a failure occurs: if the failure occurs, the failure node persistence file migration recovery algorithm ensures the oversized file transmission proxy cluster A Large file Is a normal operation of (a). In other words, when one oversized file transmission agent in the oversized file transmission agent cluster fails, the normal operation of the whole oversized file transmission agent cluster can be ensured, and the restoration of the failed oversized file transmission agent is supported to be added into the cluster again.
(1) When the oversized file transmission agent joins the cluster, the oversized file transmission agent needs to report own information to the scheduling service and periodically report own survivability through a heartbeat mechanism. The scheduling service can judge the fault of the oversized file transmission agent after the heartbeat is overtime;
(2) When an oversized file transfer agent is determined to be faulty, it will not be the target of a new transfer task. The existing transmission task re-requests and acquires a new oversized file transmission agent as a target of the transmission task after overtime and determining faults with the scheduling service;
(3) After judging the fault of the oversized file transmission agent, the scheduling service shall initiate a persistent message migration instruction to the daemon on the corresponding node, migrate the persistent message to other oversized file transmission agents for retransmission, so as to ensure the normal work of the whole oversized file transmission agent cluster. When trying to transfer the persistent file, the daemon should determine whether the persistent message is complete, if so, the daemon is transferred; if the persistent message is incomplete, it is deleted directly (waiting for the remote communication application to reinitiate the transmission). The detailed description is shown in algorithm 3.
Further, the fault judging flow is as follows:
the ultra-large file transmission agent is added into the ultra-large file transmission agent cluster A Largefile When the method is used, the information of the user is reported, and the survivability of the user is periodically reported through a heartbeat mechanism; the oversized file transfer agent (example: 3) every other heartbeat cycleSecond), a heartbeat message needs to be sent to the dispatch service. If the dispatch service does not receive the heartbeat message from the oversized file transfer service within a heartbeat check period (e.g., 10 seconds), the heartbeat of the oversized file transfer service is considered to be overtime, i.e., the oversized file transfer service is determined to be faulty.
Algorithm 3 failure node persistent file migration recovery algorithm
Input: fault node information
And (3) outputting: failure migration recovery result
The following further describes the solution and effects of the present application in conjunction with specific embodiments:
fig. 4 shows the architecture of a scheduling service, which is composed of 6 parts in total, namely a discovery module, an agent cluster management module, a transmission task response module, a scheduling connection module, a scheduling control module and a daemon management module. The discovery module is responsible for receiving and feeding back a link establishment request of remote communication; the transmission task response module is responsible for processing transmission task scheduling of the far-end communication application; the agent cluster management module is responsible for the joining, exiting and constructing global information views of oversized file transmission tasks in the whole cluster; the scheduling connection module is responsible for scheduling decisions; the scheduling control module is responsible for interacting with the daemon process when faults occur in the cluster, sending control instructions and receiving instruction execution results; the daemon management module uniformly manages the survival states of daemons in the cluster.
Fig. 5 illustrates a process for a remote communication application to establish a connection with a very large file transfer agent cluster:
the method comprises the steps that firstly, a remote communication application firstly sends a link establishment request to an external IP address and a port of an oversized file transmission proxy cluster.
Secondly, after receiving the link establishment request, the scheduling service in the oversized file transmission agent cluster carries out link establishment request feedback to feed back the IP of each oversized file transmission agent in the cluster i The address and port are fed back to the remote communication application.
Third, after receiving the feedback of the link establishment request, the remote communication application is connected with the oversized file transmission agent based on the feedback contentA transmission link is established.
Fourth, after receiving the feedback of the link establishment request, the remote communication application is connected with the oversized file transmission agent based on the feedback contentA transmission link is established.
Record forms of the remote communication application on the oversized file transfer service proxy are shown in table 1:
TABLE 1
Record forms of the oversized file transfer agent cluster on its remote communication application are shown in table 2:
TABLE 2
Fig. 6 illustrates a task scheduling process of a remote communication application when initiating a transmission task:
the method comprises the steps that a remote communication application firstly initiates a transmission task request to a scheduling service, and characteristic information of a transmission task is attached to the transmission task request.
And secondly, after receiving the transmission task request, the scheduling service calculates a transmission task destination (namely, which oversized file transmission agent is required to be selected to be responsible for the transmission task) based on the cluster global information view and the characteristic information of the transmission task.
And thirdly, the scheduling service feeds back the transmission task destination to the calculation result to the far-end communication application.
Fourth step, after receiving the feedback of the transmission task request, the remote communication application transmits the agent to the oversized fileA transmission task is initiated.
Fifth step, super large file transmission proxyAfter receiving the transmission data, the transmission data is subjected to persistent storage processing.
Sixth step, oversized file transmission proxyAfter completion of the transmission data persistence storage, the transmission data is forwarded to the transmission destination.
FIG. 7 illustrates a recovery process for a transfer task that a failed node has persisted after a failure of one of the oversized file transfer agents within the cluster:
first, the dispatch service perceives the super-large file transmission agent based on the heartbeat mechanismA failure occurs.
And secondly, the scheduling service initiates a persistent transmission task recovery instruction to the daemon 1 on the fault oversized file transmission proxy node so as to ensure that all transmission tasks are forwarded normally.
Third, the daemon 1 screens the persistent transmission tasks after receiving the instruction, and only retains and copies the complete transmission tasks.
Fourth, daemon 1 initiates a transport task migration request to the dispatch service.
Fifth, the daemon feeds back a persistent transmission task migration destination list to the daemon 1 based on the scheduling policy.
Sixth, daemon 1 migrates the persistent transfer task to the oversized file transfer agentWhere it is located.
Seventh, daemon 1 notifies the dispatch service that the persistent transmission task migration has been completed.
Eighth step, dispatch service transmits proxy to oversized fileAnd initiating a loading transmission task instruction.
Ninth step, super large file transmission proxyThe persistent transmission task in the receive buffer is loaded.
Tenth step, oversized file transfer agentThe forwarding transfer data, i.e. the transfer tasks of the persistent storage under the failed oversized file transfer agent 1, have been restored and successfully transferred.
In summary, the embodiment of the invention provides a transmission task scheduling method based on nginx, which breaks through the bottleneck problem of data storage forwarding capability of a single device by clustering oversized file transmission agents, only expands the oversized file transmission agent clusters when a certain system needs higher content data traffic, and solves the problem that the oversized file transmission agents deployed by a single node cannot meet the increasing transmission traffic demand due to the capability of hardware devices. In addition, on the basis of the transmission agent cluster, the fault tolerance and the fault recovery capability are expanded, and the robustness of a certain system is enhanced.
In addition, as shown in fig. 8, the second embodiment of the present invention further proposes a transmission task scheduling system 20 based on nginx, including, for example: a transmission link establishment module 201, a task scheduling forwarding module 202 and a failure recovery module 203.
The transmission link establishment module 201 is configured to establish communication connection with an oversized file transmission agent in the oversized file transmission agent cluster by using a reverse proxy mechanism of the nginx service; the task scheduling forwarding module 202 is configured to transmit a message to a corresponding oversized file transfer agent in the oversized file transfer agent cluster based on a transmission task scheduling policy, where the oversized file transfer agent is responsible for forwarding a task of the message and writing corresponding message information into a common database table; the failure recovery module 203 is configured to enable normal operation of the entire oversized file transfer agent cluster when an oversized file transfer agent within the oversized file transfer agent cluster fails, and to support rejoining the cluster after recovery of the failed oversized file transfer agent.
The method for quickly solving the set combination implemented by the transmission task scheduling system 20 based on the nginx according to the second embodiment of the present invention is as described in the foregoing first embodiment, and will not be described in detail herein. Optionally, each module in the second embodiment and the other operations or functions described above are respectively for implementing the method described in the first embodiment, and the beneficial effects of this embodiment are the same as those of the foregoing first embodiment, which are not described herein for brevity.
As shown in fig. 9, a third embodiment of the present invention also proposes an electronic device 30, for example, including: a memory 31 and one or more processors 32 connected to the memory 31, where the memory 31 stores a computer program, and the processors 32 are configured to execute the computer program to implement the steps of the method described in the first embodiment, and the electronic device 30 provided in this embodiment has the same advantages as the method for scheduling transmission tasks based on ngix provided in the first embodiment.
As shown in fig. 10, the fourth embodiment of the present invention further proposes a computer readable storage medium 40, which is a non-volatile memory and stores computer readable instructions, which when executed by one or more processors, for example, cause the one or more processors to perform the method for scheduling transmission tasks based on ngix described in the foregoing first embodiment. The specific method may be described with reference to the first embodiment, and for brevity, will not be described herein again, and the beneficial effects of the computer readable storage medium 40 provided in this embodiment are the same as those of the transmission task scheduling method based on nginx provided in the first embodiment.
The computer readable storage medium may include, among other things, any type of disk including floppy disks, optical disks, DVDs, CD-ROMs, micro-drives, and magneto-optical disks, ROM, RAM, EPROM, EEPROM, DRAM, VRAM, flash memory devices, magnetic or optical cards, nanosystems (including molecular memory ICs), or any type of media or device suitable for storing instructions and/or data.
It should be noted that, for simplicity of description, the foregoing method embodiments are all expressed as a series of action combinations, but it should be understood by those skilled in the art that the present application is not limited by the order of actions described, as some steps may be performed in other order or simultaneously in accordance with the present application. Further, those skilled in the art will also appreciate that the embodiments described in the specification are all preferred embodiments, and that the acts and modules referred to are not necessarily required in the present application.
In the foregoing embodiments, the descriptions of the embodiments are emphasized, and for parts of one embodiment that are not described in detail, reference may be made to related descriptions of other embodiments.
Those of ordinary skill in the art will appreciate that all or a portion of the steps in the various methods of the above embodiments may be performed by hardware associated with a program that is stored in a computer readable memory, which may include: flash disk, read-Only Memory (ROM), random-access Memory (Random Access Memory, RAM), magnetic or optical disk, and the like.
The foregoing is merely exemplary embodiments of the present disclosure and is not intended to limit the scope of the present disclosure. That is, equivalent changes and modifications are contemplated by the teachings of this disclosure, which fall within the scope of the present disclosure. Embodiments of the present disclosure will be readily apparent to those skilled in the art from consideration of the specification and practice of the disclosure herein. This application is intended to cover any adaptations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a scope and spirit of the disclosure being indicated by the claims.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
It will be readily appreciated by those skilled in the art that the foregoing description is merely a preferred embodiment of the invention and is not intended to limit the invention, but any modifications, equivalents, improvements or alternatives falling within the spirit and principles of the invention are intended to be included within the scope of the invention.

Claims (10)

1. The transmission task scheduling method based on the nginx is characterized by comprising the following steps of:
creating an oversized file transmission agent cluster based on the oversized file transmission agent, and establishing communication connection with each oversized file transmission agent in the oversized file transmission agent cluster by utilizing a reverse agent mechanism of the nginx service;
sensing state information of an oversized file transmission agent in the oversized file transmission agent cluster, taking the state information as a decision basis of a transmission task scheduling strategy, and combining message characteristics to obtain a decision result;
transmitting the message to a corresponding oversized file transmission agent in the oversized file transmission agent cluster according to the decision result, wherein the oversized file transmission agent is responsible for forwarding the message and writing the corresponding message information into a shared database table;
whether the oversized file transmission agent in the oversized file transmission agent cluster breaks down or not is perceived in real time, and if so, the fault node persists the file migration recovery algorithm, so that the normal operation of the oversized file transmission agent cluster is ensured.
2. The method for scheduling transmission tasks based on nginx according to claim 1, wherein said establishing a communication connection with each oversized file transmission agent in the oversized file transmission agent cluster comprises:
the method comprises the steps that a link establishment request is sent to an external IP address and a port of an oversized file transmission proxy cluster, and the link establishment request is waited to be fed back, if waiting time is longer than a preset time length, the link establishment fails; otherwise, acquiring IP of all oversized file transfer agents in the oversized file transfer agent cluster i Address and port set, traversing each IP in the port set i Address and try IP i The address establishes a remote communication link, if the link still fails to be established, the IP is used for the link establishment i Adding the address into a retry address set;
when the retry address set is a non-empty set, initiating a link establishment confirmation, and confirming the IP i Adding the address into the port set again, and emptying the retry address set;
repeating the above process until all the IPs i The address successfully establishes the communication link.
3. The nginx based transmission task scheduling method of claim 2, further comprising:
and after the ultra-large file transmission proxy cluster is updated, updating the communication connection based on the new ultra-large file transmission proxy cluster when a new message transmission task is initiated.
4. The rginx based transmission task scheduling method according to claim 1, wherein said status information comprises:
static information representing the joining state of the oversized file transfer agent when the oversized file transfer agent is added into the oversized file transfer agent cluster; and dynamic information characterizing the operational state of the oversized file transfer agent when running.
5. The method for scheduling transmission tasks based on nginx according to claim 4, wherein said decision result obtaining process comprises:
presetting reference weight values of different operation parameters n of the oversized file transfer agent and using a set P i The representation, wherein,
generating a set of motion state descriptions based on dynamic informationWherein (1)>A numerical value representing different operating parameters n of the oversized file transfer agent;
acquiring a message x running state weight adjustment value set P x ，Wherein,representing the message x for different operating parameters x n Is a set of the adjustment values of (2);
when a message x is sent, calculating the running state comprehensive weight of the oversized file transmission agent as follows:
comprehensive weight based on running stateThe size of the super-large file transmission agent of the message x is randomly determined, and the running state comprehensive weight of the super-large file transmission agent is +.>Probability PR of becoming the very large file transfer agent i The calculated relation of (2) is: />
6. The nginx based transmission task scheduling method of claim 1, wherein the failed node persistent file migration recovery algorithm comprises:
marking the failed oversized file transfer agent as a failed oversized file transfer agent, wherein the failed oversized file transfer agent is no longer a target of a transfer task;
initiating a persistent file migration instruction to a daemon on a fault oversized file transfer agent node, transferring the file which is completely transferred to the fault oversized file transfer agent node to other oversized file transfer agent nodes, and taking charge of forwarding the message by other oversized file transfer agent nodes.
7. The method for scheduling transmission tasks based on nginx according to claim 1, wherein the fault judging process comprises:
when the oversized file transmission agent joins the oversized file transmission agent cluster, reporting self information and periodically reporting the survivability of the oversized file transmission agent through a heartbeat mechanism;
and sending a heartbeat message by the oversized file transmission agent every preset heartbeat period, and if the heartbeat message from the oversized file transmission agent is not received in a preset heartbeat check period, regarding the oversized file transmission agent as a heartbeat timeout, and judging that the oversized file transmission agent is a transmission service fault.
8. A transmission task scheduling system based on nginx, configured to implement a method according to any one of claims 1 to 7, comprising:
the transmission link establishment module is used for establishing communication connection with the oversized file transmission agent in the oversized file transmission agent cluster by utilizing a reverse agent mechanism of the nginx service;
the task scheduling and forwarding module is used for transmitting the message to a corresponding oversized file transmission agent in the oversized file transmission agent cluster based on a transmission task scheduling strategy, and the oversized file transmission agent is responsible for forwarding the message and writing the corresponding message information into a shared database table;
the fault recovery module is arranged to enable the whole oversized file transmission agent cluster to normally operate when the oversized file transmission agent in the oversized file transmission agent cluster breaks down, and support the restoration of the broken oversized file transmission agent and then rejoin the cluster.
9. An electronic device, comprising: a memory and one or more processors coupled to the memory, the memory storing a computer program for executing the computer program to perform the steps of the method of any of claims 1 to 7.
10. A computer readable storage medium storing computer executable instructions for performing the steps of the method according to any one of claims 1 to 7.
CN202410064418.2A 2024-01-16 2024-01-16 Transmission task scheduling method and application based on nginx Pending CN117880276A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202410064418.2A CN117880276A (en) 2024-01-16 2024-01-16 Transmission task scheduling method and application based on nginx

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202410064418.2A CN117880276A (en) 2024-01-16 2024-01-16 Transmission task scheduling method and application based on nginx

Publications (1)

Publication Number Publication Date
CN117880276A true CN117880276A (en) 2024-04-12

Family

ID=90580842

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202410064418.2A Pending CN117880276A (en) 2024-01-16 2024-01-16 Transmission task scheduling method and application based on nginx

Country Status (1)

Country Link
CN (1) CN117880276A (en)

Similar Documents

Publication Publication Date Title
US6977908B2 (en) Method and apparatus for discovering computer systems in a distributed multi-system cluster
US9577920B2 (en) Communication system, control device, node, processing rule setting method and program
CN111615066B (en) Distributed micro-service registration and calling method based on broadcast
US8375001B2 (en) Master monitoring mechanism for a geographical distributed database
JP5246157B2 (en) Load balancing system
KR101099822B1 (en) Redundant routing capabilities for a network node cluster
US6061807A (en) Methods systems and computer products for error recovery of endpoint nodes
CN110224871A (en) A kind of high availability method and device of Redis cluster
JP2004519024A (en) System and method for managing a cluster containing multiple nodes
JP4808731B2 (en) Method for adjusting load between subsystems in a communication network system
CN107948063B (en) Method for establishing aggregation link and access equipment
JP4964666B2 (en) Computer, program and method for switching redundant communication paths
US20100082822A1 (en) Technique for realizing high reliability in inter-application communication
JP5408620B2 (en) Data distribution management system and data distribution management method
CN112615901B (en) Method for sending user request by client and storage system
JP5526780B2 (en) Load distribution system, service processing server, load distribution method, and load distribution program
CN110351122B (en) Disaster recovery method, device, system and electronic equipment
CN117880276A (en) Transmission task scheduling method and application based on nginx
CN109688011B (en) Agent selection method and device based on OpenStack
CN114095342B (en) Backup realization method and device
CN114598593B (en) Message processing method, system, computing device and computer storage medium
US20050182763A1 (en) Apparatus and method for on-line upgrade using proxy objects in server nodes
US10536875B2 (en) System and method for seamless TCP connection handoff
JP5228369B2 (en) COMMUNICATION SYSTEM, COMMUNICATION METHOD, AND COMMUNICATION PROGRAM
US11947431B1 (en) Replication data facility failure detection and failover automation

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination