CN114489995A - Distributed scheduling processing method and system - Google Patents
Distributed scheduling processing method and system Download PDFInfo
- Publication number
- CN114489995A CN114489995A CN202210139075.2A CN202210139075A CN114489995A CN 114489995 A CN114489995 A CN 114489995A CN 202210139075 A CN202210139075 A CN 202210139075A CN 114489995 A CN114489995 A CN 114489995A
- Authority
- CN
- China
- Prior art keywords
- node
- task
- connection
- control end
- main control
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/50—Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems
- G06F21/57—Certifying or maintaining trusted computer platforms, e.g. secure boots or power-downs, version controls, system software checks, secure updates or assessing vulnerabilities
- G06F21/577—Assessing vulnerabilities and evaluating computer system security
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1441—Countermeasures against malicious traffic
- H04L63/1491—Countermeasures against malicious traffic using deception as countermeasure, e.g. honeypots, honeynets, decoys or entrapment
Landscapes
- Engineering & Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Computer Hardware Design (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer And Data Communications (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
A distributed scheduling processing method and system, obtain the node information of task that the front end transmits through the top management, copy the task execution script to the node machine and run and start the script; when the node is successfully established, configuration information is obtained from a database through a main control end, socket connection is sequentially established with the node, a data packet with a connection password is sent, and heartbeat is established; the node receives the data packet with the connection password, and analyzes and checks the data packet with the connection password; if the master control end successfully issues the task execution instruction, the node splits and analyzes the structural body information transmitted by the master control end; the node establishes a long remote call connection with the main control end and creates a task queue and a result queue. The distributed task scheduling mechanism is adopted, the data collection capacity is improved, the distributed task nodes cannot store main control traces, and the backtracking risk is reduced to the greatest extent; by adopting a distributed detection mode, the target data volume can be filtered and subjected to cross duplicate removal detection, and the vulnerability detection efficiency is improved.
Description
Technical Field
The invention relates to a distributed scheduling processing method and a distributed scheduling processing system, and belongs to the technical field of automatic vulnerability data processing.
Background
In the existing distributed task queue design, a main control end controls task process issuing through a BaseBanger (asynchronous process control method), distributed nodes execute, and results are asynchronously returned, so that the task execution efficiency is improved, and the risk of exposure of the main control end is reduced. The task scheduling execution mechanism adopts an APScheduler (composite task scheduling framework) scheduling framework to determine the sustainable periodic call of a scheduling object, and sets an offline and retry mechanism, thereby avoiding the problem that the task issuing execution period cannot be triggered due to system abnormity and ensuring the stability of task scheduling triggering to the maximum extent.
In the prior art, based on the distributed task queue design, if the task queue is in a reverse connection mode, the connection address of the main control end is easily exposed, and the main control backtracking security risk is generated. Based on the existing data collection function, there is no way to collect all data on the internet, and differences are inevitably generated in the number of whole-network vulnerability verifications. Based on the existing vulnerability detection function, the problems of large detection data volume, long time consumption and low detection accuracy rate exist.
Disclosure of Invention
Therefore, the invention provides a distributed scheduling processing method and a distributed scheduling processing system, which solve the problems that a distributed task node stores a main control trace, the main control backtracking risk exists, and the task is easy to be abnormal.
In order to achieve the above purpose, the invention provides the following technical scheme: a distributed scheduling processing method comprises the following steps:
node deployment: task node information transmitted by the front end is acquired through the main control end, a task execution script is copied to a node machine to run, and the script is started;
and (3) node connection: when the node is successfully established, configuration information is obtained from a database through a main control end, socket connection is sequentially established with the node, a data packet with a connection password is sent, and heartbeat is established;
password verification: the node receives the data packet with the connection password, and analyzes and checks the data packet with the connection password;
executing the instructions: if the master control end successfully issues the task execution instruction, the node splits and analyzes the structural body information transmitted by the master control end; the node establishes a long remote call connection with the main control end and creates a task queue and a result queue.
As a preferred scheme of the distributed scheduling processing method, in the process of node deployment, the task node information comprises a node address, a node port and a connection password, and the nodes are connected through the connection password;
if the script runs successfully, the node address, the connection port and the connection password state are stored through the main control end, and connection is waited; if the script fails to operate, returning a message of failed node deployment to the front end;
in the process of node connection, the configuration information acquired from the database through the main control end comprises a node address, a node port and a connection password.
As a preferred scheme of the distributed scheduling processing method, in the password verification process, if the password is wrong, a mark for rejecting connection is sent to the main control end, the connection is actively disconnected, and the heartbeat is stopped; if the password is correct, a sign that the password passes the verification is sent to the main control end, and the command interaction sent by the main control end is waited to be received.
As a preferred scheme of the distributed scheduling processing method, in the instruction execution process, analyzing structure information of the master control end comprises a connection address, a connection port, a connection key, function library dependence, an application identification fingerprint and a task execution script;
in the instruction execution process, the main control end puts tasks into a task queue, the nodes read the tasks from the task queue and send the read tasks to an asynchronous task executor object;
and after all task instructions are executed, the recorded task results are put into a result queue, and the main control end acquires the task results from the result queue and stores the task results into a database.
As a preferred scheme of the distributed scheduling processing method, the execution action of the asynchronous task executor comprises data collection, the task executor receives a data collection task, and after parameters are extracted, the nodes send the data to a space search engine according to parameter configuration to obtain matching data;
after the data collection task is completed, the main control end puts the result of the data collection task into a task queue again to be used as the input of the vulnerability detection task;
the node acquires the task from the task queue again, and then executes the host detection action: and the task executor receives the vulnerability task and transmits a data collection result as a unique data source of the vulnerability.
As a preferred scheme of the distributed scheduling processing method, the execution action of the asynchronous task executor further comprises application identification, an application identification algorithm built in the task executor is called, and application identification is carried out by combining an application identification fingerprint transmitted by the structural body.
As the preferred scheme of the distributed scheduling processing method, the execution action of the asynchronous task executor also comprises vulnerability verification, when the application identification is successful, a vulnerability verification PoC frame is arranged in the node, the node initiates a request with a carrier to a target website, and the verification state is judged according to the target response;
and when the vulnerability verification is successful, calling a vulnerability utilization algorithm of the task executor, initiating a request to a target website by the node, and judging whether the vulnerability utilization is successful according to a target response.
As a preferred scheme of the distributed scheduling processing method, the method further includes that the destruction node: and when the master control end issues the node destruction instruction, the node finishes the self process, deletes the self configuration and carries out self destruction.
The invention also discloses a distributed scheduling processing system, which adopts the distributed scheduling processing method and comprises the following steps:
the node deployment unit is used for acquiring task node information transmitted by the front end through the main control end, copying a task execution script to a node machine for running and starting the script;
the node connection unit is used for acquiring configuration information from the database through the main control end when the node is successfully established, sequentially establishing socket connection with the node, sending a data packet with a connection password and establishing heartbeat;
the password checking unit is used for analyzing and checking the data packet with the connection password after the node receives the data packet with the connection password;
the execution instruction unit is used for splitting and analyzing the structural body information transmitted by the main control end by the node if the main control end successfully issues the task execution instruction; the node establishes a long remote call connection with the main control end and creates a task queue and a result queue.
As a preferred scheme of the distributed scheduling processing system, in a node deployment unit, the task node information includes a node address, a node port and a connection password, and the nodes are connected through the connection password;
if the script runs successfully, the node address, the connection port and the connection password state are stored through the main control end, and connection is waited; if the script fails to run, a message of failed node deployment is returned to the front end.
As a preferred scheme of the distributed scheduling processing system, in the node connection unit, the configuration information acquired from the database through the main control terminal includes a node address, a node port and a connection password.
As a preferred scheme of the distributed scheduling processing system, in the execution instruction unit, the structural body information analyzed by the main control end comprises a connection address, a connection port, a connection key, function library dependence, an application identification fingerprint and a task execution script;
in the execution instruction unit, a main control end puts tasks into a task queue, a node reads the tasks from the task queue and sends the read tasks to an asynchronous task executor object;
and after all task instructions are executed, the recorded task results are put into a result queue, and the main control end acquires the task results from the result queue and stores the task results into a database.
As a preferred scheme of the distributed scheduling processing system, the execution instruction unit comprises a data collection subunit, and the data collection subunit is used for receiving a data collection task by a task executor, extracting parameters, and sending the parameters to a space search engine by a node according to parameter configuration to obtain matching data; after the data collection task is completed, the main control end puts the result of the data collection task into a task queue again to be used as the input of the vulnerability detection task;
the system also comprises a host detection subunit, which is used for the node to acquire the task from the task queue again and then execute the host detection action: and the task executor receives the vulnerability task and transmits a data collection result as a unique data source of the vulnerability.
As a preferred scheme of the distributed scheduling processing system, the execution instruction unit further includes an application identification subunit, which is used for calling an application identification algorithm built in the task executor and performing application identification by combining an application identification fingerprint transmitted from the structure body.
As a preferred scheme of the distributed scheduling processing system, the instruction execution unit further includes a vulnerability verification subunit, which is used for building a vulnerability verification PoC framework in the node when the application identification is successful, the node initiates a request with a carrier to the target website, and the verification state is judged according to the target response.
As a preferred scheme of the distributed scheduling processing system, the instruction execution unit further includes a vulnerability exploiting subunit, which is used for calling a vulnerability exploiting algorithm of the task executor when vulnerability verification is successful, and the node initiates a request to the target website and judges whether vulnerability exploitation is successful according to the target response.
As a preferred scheme of the distributed scheduling processing system, the distributed scheduling processing system further includes a destroy node unit, which is used for ending the self-process by the node and deleting the self-configuration for self-destruction when the main control end issues a node destroy instruction.
The invention has the following advantages: task node information transmitted by the front end is acquired through the main control end, a task execution script is copied to a node machine to run, and the script is started; when the node is successfully established, configuration information is obtained from a database through a main control end, socket connection is sequentially established with the node, a data packet with a connection password is sent, and heartbeat is established; the node receives the data packet with the connection password, and analyzes and checks the data packet with the connection password; if the master control end successfully issues the task execution instruction, the node splits and analyzes the structural body information transmitted by the master control end; the node establishes a long remote call connection with the main control end and creates a task queue and a result queue. According to the invention, a distributed task scheduling mechanism is adopted, so that the data collection capacity can be improved, the progress of batch vulnerability verification on assets on the Internet is accelerated, the global vulnerability distribution condition is sensed in real time, the probability of abnormity of tasks is reduced, the distributed task nodes cannot store main control traces, and the main control backtracking risk is reduced to the greatest extent; by adopting a distributed detection mode, the target data volume can be filtered and subjected to cross duplicate removal detection, and the vulnerability detection efficiency is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below. It should be apparent that the drawings in the following description are merely exemplary and that other implementation drawings may be derived from the provided drawings by those of ordinary skill in the art without inventive effort.
The structures, ratios, sizes, and the like shown in the present specification are only used for matching with the contents disclosed in the specification, so that those skilled in the art can understand and read the present invention, and do not limit the conditions for implementing the present invention, so that the present invention has no technical significance, and any structural modifications, changes in the ratio relationship, or adjustments of the sizes, without affecting the functions and purposes of the present invention, should still fall within the scope of the present invention.
Fig. 1 is a schematic flow chart of a distributed scheduling processing method provided in an embodiment of the present invention;
fig. 2 is a schematic technical route of a distributed scheduling processing method according to an embodiment of the present invention;
fig. 3 is a schematic diagram of a distributed scheduling processing system provided in an embodiment of the present invention.
Detailed Description
The present invention is described in terms of particular embodiments, other advantages and features of the invention will become apparent to those skilled in the art from the following disclosure, and it is to be understood that the described embodiments are merely exemplary of the invention and that it is not intended to limit the invention to the particular embodiments disclosed. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Example 1
Referring to fig. 1 and 2, embodiment 1 of the present invention provides a distributed scheduling processing method, including the following steps:
s1, node deployment: task node information transmitted by the front end is acquired through the main control end, a task execution script is copied to a node machine to run, and the script is started;
s2, connecting nodes: when the node is successfully established, configuration information is obtained from a database through a main control end, socket connection is sequentially established with the node, a data packet with a connection password is sent, and heartbeat is established;
s3, password verification: the node receives the data packet with the connection password, and analyzes and checks the data packet with the connection password;
s4, executing the instruction: if the master control end successfully issues the task execution instruction, the node splits and analyzes the structural body information transmitted by the master control end; the node establishes a long remote call connection with the main control end and creates a task queue and a result queue.
In this embodiment, in the step S1, in the process of node deployment, the task node information includes a node address, a node port, and a connection password, and the nodes are connected by the connection password;
if the script runs successfully, the node address, the connection port and the connection password state are stored through the main control end, and connection is waited; if the script fails to run, a message of failed node deployment is returned to the front end.
Specifically, the main control end obtains task node information transmitted by the front end, the task node information comprises a node address, a connection port and a connection password, the task execution script is remotely copied to a node machine through the connection password connection node, a configuration file is modified, an occupation port and the connection password are set, and then the start script is operated. If the script runs successfully, the master control end stores the node address, the connection port and the connection password state and waits for connection; if the script fails to run, a message of failed deployment of the node is returned to the front end.
In this embodiment, in the step S2, in the process of connecting the nodes, the configuration information obtained from the database through the main control end includes a node address, a node port, and a connection password.
Specifically, if the node is successfully created, the main control end obtains configuration information from the database: the node address, the connection port and the connection password are sequentially connected with the node through a socket, a data packet with the connection password is sent, and heartbeat is established.
In this embodiment, in the password verification process in step S3, if the password is wrong, a connection rejection flag is sent to the main control end, and the connection is actively disconnected, and the heartbeat is stopped; if the password is correct, a sign that the password passes the verification is sent to the main control end, and the command interaction sent by the main control end is waited to be received.
Specifically, when a node receives a data packet with a connection password, the security and the correctness are analyzed and verified, if the connection password is wrong, a mark for rejecting connection is sent to a main control end, the connection is actively disconnected, and the heartbeat is stopped; if the connection password is correct, a mark that the password passes the verification is sent to the main control end, and the command interaction issued by the main control end is waited to be received.
In this embodiment, in the process of executing the instruction in step S4, analyzing the structure information of the master control end includes a connection address, a connection port, a connection key, a function library dependency, an application identification fingerprint, and a task execution script;
in the instruction execution process, the main control end puts tasks into a task queue, the nodes read the tasks from the task queue and send the read tasks to an asynchronous task executor object;
and after all task instructions are executed, the recorded task results are put into a result queue, and the main control end acquires the task results from the result queue and stores the task results into a database.
Specifically, if the master control terminal successfully issues the task execution instruction, the node automatically disassembles the transmitted structural body, and analyzes the structural body information of the master control terminal: node address, connection port, connection key, function library dependence, application identification fingerprint, and execution script. Firstly, installing function library dependence required by an execution script, if the function library dependence is successful, registering an execution code into a task execution framework of a node, and starting to generate a task executor object; and if the module fails to be installed, sending a message of module installation failure to the main control end, and actively disconnecting the connection with the main control end. After this step is completed, the node will establish a long remote call connection with the master and create two message queues: a task queue and a result queue. The master control end puts the tasks into the task queue, the nodes read the tasks from the task queue and send the tasks to the asynchronous task executor object, and the execution of the task executor object has strict pipeline sequence flow.
In the embodiment, the execution action of the asynchronous task executor comprises data collection, the task executor receives a data collection task, and after parameters are extracted, the nodes send the data to a space search engine according to parameter configuration to obtain matching data; and after the data collection task is completed, the main control end puts the result of the data collection task into the task queue again to be used as the input of the vulnerability detection task.
Specifically, after a task executor receives a data collection task and extracts parameters (query parameters and a query engine), a node sends the parameters to a space search engine to obtain matching data according to parameter configuration, and if the number of the obtained data is less than 1 ten thousand, a data packet is directly returned; if the quantity of the acquired data is more than 1 ten thousand, the data collection main process can be recombined into a plurality of query sub-grammar processes according to the national (regional) characteristics, and a mark is generated and placed in a task queue; if the network request is abnormal or an abnormal message is returned, the sub-grammar task mark is placed in the task queue again to wait, the request is initiated again after the task is triggered and randomly delayed for 1-7 seconds until the time exceeds the request limit threshold value for 30 times, and finally the abnormal message is returned. Finally, the data collection host process will pull the flags of the sub-queries to aggregate into a complete data packet.
And after the data collection task is completed, the main control end puts the result of the data collection task into the task queue again to be used as the input of the vulnerability detection task.
In this embodiment, the node acquires the task from the task queue again, and then executes a host detection action: and the task executor receives the vulnerability task and transmits a data collection result as a unique data source of the vulnerability.
Specifically, the task executor receives the exploit task and transmits a data collection flag to serve as a unique data source of the exploit. And calling a host detection algorithm of the task executor, establishing a connection probe between the node and the target port, and judging the opening condition of the target port. If the port is not opened, returning a host exception status code: 0x0008 and exit the detection of the target; if the honeypot is opened, honeypot identification is carried out, honeypot identification detection realizes detection and judgment through keywords and page local features, and if the honeypot is a honeypot, a suspected honeypot state code is returned: 0x0004, and exit the task executor; if not, returning to the normal state code of the host: 0x 0002. Then proceed to the next step: application identification
In this embodiment, the execution action of the asynchronous task executor further includes application identification, an application identification algorithm built in the task executor is called, and application identification is performed by combining an application identification fingerprint transmitted from the structural body.
Specifically, an application identification algorithm is built in the task executor, and application identification is carried out by combining the application identification fingerprint transmitted by the structural body. Firstly, the node simulates to visit a target site, and matches a returned data response body with an applied fingerprint rule. If the fingerprints are not matched successfully, returning an identification failure status code: 0x0040 and close the task executor; if the matching is successful, returning an identification success status code: 0x 0020; in addition, in order to ensure the stability of the application identification process, the task executor throws an identification abnormal state code when executing the abnormality: 0x0080, exit with it and record the result. Then the next step is carried out: and (5) vulnerability verification.
In this embodiment, the execution action of the asynchronous task executor further includes vulnerability verification, when the application identification is successful, a vulnerability verification PoC framework is built in the node, the node initiates a request with a bearer to a target website, and the verification state is judged according to a target response.
Specifically, the application identifies that a successful task will perform the operation of vulnerability verification. The node is internally provided with a vulnerability verification PoC frame, the node can initiate a request (a detection mode) with a special load to a target website, the verification state is judged according to a target response, and if the verification fails, a verification failure state code is returned: 0x0400, and close the task executor; if the verification is successful, returning a verification success status code: 0x 0200; if the verification framework has problems or the script execution has unexpected exceptions (node IP is forbidden, etc.), a verification exception status code is returned: 0x0800, followed by exit and recording of the results. And then carrying out next vulnerability exploiting operation.
In this embodiment, the execution action of the asynchronous task executor further includes vulnerability exploitation, when vulnerability verification is successful, a vulnerability exploitation algorithm of the task executor is called, the node initiates a request to the target website, and whether vulnerability exploitation is successful is judged according to the target response.
Specifically, the target that is successfully verified will automatically begin the instruction operations of the exploit. Calling a vulnerability exploitation algorithm of a task executor, initiating a specific request to a target website by a node, judging whether exploitation is successful according to a target response, and if the exploitation is failed, returning an exploitation failure status code: 0x 4000; if the utilization is successful, returning a utilization success status code: 0x2000, and return data after success is utilized.
In this embodiment, the method further includes step S5, where the destroy node: and when the master control end issues the node destruction instruction, the node finishes the self process, deletes the self configuration and carries out self destruction.
In summary, the task node information transmitted by the front end is acquired by the main control end, the task execution script is copied to the node machine to run, and the script is started; when the node is successfully established, configuration information is obtained from a database through a main control end, socket connection is sequentially established with the node, a data packet with a connection password is sent, and heartbeat is established; the node receives the data packet with the connection password, and analyzes and checks the data packet with the connection password; if the master control end successfully issues the task execution instruction, the node splits and analyzes the structural body information transmitted by the master control end; the node establishes a long remote call connection with the main control end and creates a task queue and a result queue. The execution action of the asynchronous task executor comprises data collection, the task executor receives a data collection task, and after parameters are extracted, the nodes send the data to a space search engine according to parameter configuration to obtain matching data; and after the data collection task is completed, the main control end puts the result of the data collection task into the task queue again to be used as the input of the vulnerability detection task. And after the data collection task is completed, the main control end puts the result of the data collection task into the task queue again to be used as the input of the vulnerability detection task. The node acquires the task from the task queue again, and then executes a host detection action: and the task executor receives the vulnerability task and transmits a data collection result as a unique data source of the vulnerability. The execution action of the asynchronous task executor also comprises application identification, an application identification algorithm built in the task executor is called, and the application identification is carried out by combining the application identification fingerprint transmitted by the structural body. And the execution action of the asynchronous task executor also comprises vulnerability verification, when the application identification is successful, a vulnerability verification PoC frame is arranged in the node, the node initiates a request with a carrier to a target website, and the verification state is judged according to the target response. And when the vulnerability verification is successful, calling a vulnerability utilization algorithm of the task executor, initiating a request to a target website by the node, and judging whether the vulnerability utilization is successful according to a target response. And when the master control end issues the node destruction instruction, the node finishes the self process, deletes the self configuration and carries out self destruction. According to the invention, a distributed task scheduling mechanism is adopted, so that the data collection capacity can be improved, the progress of batch vulnerability verification on assets on the Internet is accelerated, the global vulnerability distribution condition is sensed in real time, the probability of abnormity of tasks is reduced, the distributed task nodes cannot store main control traces, and the main control backtracking risk is reduced to the greatest extent; by adopting a distributed detection mode, the target data volume can be filtered and subjected to cross duplicate removal detection, and the vulnerability detection efficiency is improved.
Example 2
Referring to fig. 3, an embodiment 2 of the present invention further provides a distributed scheduling processing system, where a distributed scheduling processing method according to embodiment 1 or any possible implementation manner thereof is adopted, and includes:
the node deployment unit 1 is used for acquiring task node information transmitted by a front end through a main control end, copying a task execution script to a node machine for running and starting the script;
the node connection unit 2 is used for acquiring configuration information from a database through a main control end when the node is successfully established, sequentially establishing socket connection with the node, sending a data packet with a connection password and establishing heartbeat;
the password checking unit 3 is used for analyzing and checking the data packet with the connection password after the node receives the data packet with the connection password;
the execution instruction unit 4 is used for splitting and analyzing the structural body information transmitted by the main control end by the node if the main control end successfully issues the task execution instruction; the node establishes a long remote call connection with the main control end and creates a task queue and a result queue.
In this embodiment, in a node deployment unit 1, the task node information includes a node address, a node port, and a connection password, and is connected to a node by the connection password;
if the script runs successfully, the node address, the connection port and the connection password state are stored through the main control end, and connection is waited; if the script fails to run, a message of failed node deployment is returned to the front end.
In this embodiment, in the node connection unit 2, the configuration information acquired from the database through the main control end includes a node address, a node port, and a connection password.
In this embodiment, in the instruction execution unit 4, analyzing the structure information of the master control end includes a connection address, a connection port, a connection key, a function library dependency, an application identification fingerprint, and a task execution script;
in the execution instruction unit 4, a main control end puts tasks into a task queue, a node reads the tasks from the task queue and sends the read tasks to an asynchronous task executor object;
and after all task instructions are executed, the recorded task results are put into a result queue, and the main control end acquires the task results from the result queue and stores the task results into a database.
In this embodiment, the instruction execution unit 4 includes a data collection subunit 41, configured to enable the task executor to receive a data collection task, and after extracting parameters, the node sends the data collection task to the spatial search engine according to parameter configuration to obtain matching data; after the data collection task is completed, the main control end puts the result of the data collection task into a task queue again to be used as the input of the vulnerability detection task;
also included is a host detection subunit 42, configured to retrieve the task from the task queue by the node, and then perform a host detection action: and the task executor receives the vulnerability task and transmits a data collection result as a unique data source of the vulnerability.
In this embodiment, the instruction execution unit 4 further includes an application identification subunit 43, configured to invoke an application identification algorithm built in the task executor, and perform application identification by combining an application identification fingerprint transmitted from the structure.
In this embodiment, the instruction execution unit 4 further includes a vulnerability verification subunit 44, configured to, when the application identification is successful, embed a vulnerability verification PoC framework in the node, initiate a request with a bearer to the target website by the node, and determine a verification state according to the target response.
In this embodiment, the instruction execution unit 4 further includes a exploit sub-unit 45, configured to, when the vulnerability verification is successful, invoke an exploit algorithm of the task executor, and the node issues a request to the target website to determine whether the exploit is successful according to the target response.
In this embodiment, the system further includes a destruction node unit 5, configured to, when the master control end issues a node destruction instruction, end the node process, and delete the self configuration to perform self destruction.
It should be noted that, because the contents of information interaction, execution process, and the like between the units of the system are based on the same concept as the method embodiment in embodiment 1 of the present application, the technical effect brought by the contents is the same as the method embodiment of the present application, and specific contents may refer to the description in the foregoing method embodiment of the present application, and are not described herein again.
Example 3
The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
Example 4
An embodiment 4 of the present invention provides an electronic device, including: a memory and a processor;
the processor and the memory are communicated with each other through a bus; the memory stores program instructions executable by the processor, which when invoked by the processor is capable of performing the distributed scheduling processing method of embodiment 1 or any possible implementation thereof.
Specifically, the processor may be implemented by hardware or software, and when implemented by hardware, the processor may be a logic circuit, an integrated circuit, or the like; when implemented in software, the processor may be a general-purpose processor implemented by reading software code stored in a memory, which may be integrated in the processor, located external to the processor, or stand-alone.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the invention to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.).
It will be apparent to those skilled in the art that the modules or steps of the present invention described above may be implemented by a general purpose computing device, they may be centralized on a single computing device or distributed across a network of multiple computing devices, and alternatively, they may be implemented by program code executable by a computing device, such that they may be stored in a storage device and executed by a computing device, and in some cases, the steps shown or described may be performed in an order different than that described herein, or they may be separately fabricated into individual integrated circuit modules, or multiple ones of them may be fabricated into a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.
Although the invention has been described in detail above with reference to a general description and specific examples, it will be apparent to one skilled in the art that modifications or improvements may be made thereto based on the invention. Accordingly, such modifications and improvements are intended to be within the scope of the invention as claimed.
Claims (10)
1. A distributed scheduling processing method is characterized by comprising the following steps:
node deployment: task node information transmitted by the front end is acquired through the main control end, a task execution script is copied to a node machine to run, and the script is started;
and (3) node connection: when the node is successfully established, configuration information is obtained from a database through a main control end, socket connection is sequentially established with the node, a data packet with a connection password is sent, and heartbeat is established;
password verification: the node receives the data packet with the connection password, and analyzes and checks the data packet with the connection password;
executing the instructions: if the master control end successfully issues the task execution instruction, the node splits and analyzes the structural body information transmitted by the master control end; the node establishes a long remote call connection with the main control end and creates a task queue and a result queue.
2. The distributed scheduling processing method of claim 1 wherein, in the process of node deployment, the task node information includes node addresses, node ports and connection passwords, and the nodes are connected by the connection passwords;
if the script runs successfully, the node address, the connection port and the connection password state are stored through the main control end, and connection is waited; if the script fails to operate, returning a message of failed node deployment to the front end;
in the process of node connection, the configuration information acquired from the database through the main control end comprises a node address, a node port and a connection password.
3. The distributed scheduling processing method of claim 1, wherein in the password verification process, if the password is wrong, a flag for rejecting connection is sent to the master control end, the connection is actively disconnected, and the heartbeat is stopped; if the password is correct, a sign that the password passes the verification is sent to the main control end, and the command interaction sent by the main control end is waited to be received.
4. The distributed scheduling processing method of claim 1, wherein in the process of executing the instruction, analyzing structure information of the master control end includes a connection address, a connection port, a connection key, a function library dependency, an application identification fingerprint, and a task execution script;
in the instruction execution process, the main control end puts tasks into a task queue, the nodes read the tasks from the task queue and send the read tasks to an asynchronous task executor object;
and after all task instructions are executed, the recorded task results are put into a result queue, and the main control end acquires the task results from the result queue and stores the task results into a database.
5. The distributed scheduling processing method according to claim 4, wherein the execution action of the asynchronous task executor comprises data collection, the task executor receives a data collection task, and after the parameters are extracted, the nodes send the data collection task to the spatial search engine according to the parameter configuration to obtain matching data;
after the data collection task is completed, the main control end puts the result of the data collection task into a task queue again to be used as the input of the vulnerability detection task;
the node acquires the task from the task queue again, and then executes a host detection action: and the task executor receives the vulnerability task and transmits a data collection result as a unique data source of the vulnerability.
6. The distributed scheduling processing method of claim 5 wherein the execution of the asynchronous task executor further comprises application recognition, invoking an application recognition algorithm built in the task executor, and performing application recognition in combination with an application recognition fingerprint transmitted from the structure.
7. The distributed scheduling processing method of claim 6 wherein the execution of the asynchronous task executor further comprises vulnerability verification, when the application identification is successful, the node is built in a vulnerability verification PoC framework, the node initiates a request with a carrier to the target website, and determines a verification status according to the target response;
and when the vulnerability verification is successful, calling a vulnerability utilization algorithm of the task executor, initiating a request to a target website by the node, and judging whether the vulnerability utilization is successful according to a target response.
8. The distributed scheduling processing method according to claim 1, further comprising a destruction node: and when the master control end issues the node destruction instruction, the node finishes the self process, deletes the self configuration and carries out self destruction.
9. A distributed scheduling processing system using the distributed scheduling processing method according to any one of claims 1 to 8, comprising:
the node deployment unit is used for acquiring task node information transmitted by the front end through the main control end, copying a task execution script to a node machine for running and starting the script;
the node connection unit is used for acquiring configuration information from the database through the main control end when the node is successfully established, sequentially establishing socket connection with the node, sending a data packet with a connection password and establishing heartbeat;
the password checking unit is used for analyzing and checking the data packet with the connection password after the node receives the data packet with the connection password;
the execution instruction unit is used for splitting and analyzing the structural body information transmitted by the main control end by the node if the main control end successfully issues the task execution instruction; the node establishes a long remote call connection with the main control end and creates a task queue and a result queue.
10. The distributed dispatch processing system of claim 9,
in the node deployment unit, the task node information comprises a node address, a node port and a connection password, and the nodes are connected through the connection password;
if the script runs successfully, the node address, the connection port and the connection password state are stored through the main control end, and connection is waited; if the script fails to operate, returning a message of failed node deployment to the front end;
in the node connection unit, configuration information acquired from a database through a main control terminal comprises a node address, a node port and a connection password;
in the execution instruction unit, analyzing structure information of the main control end, wherein the structure information comprises a connection address, a connection port, a connection key, function library dependence, an application identification fingerprint and a task execution script;
in the execution instruction unit, a main control end puts tasks into a task queue, a node reads the tasks from the task queue and sends the read tasks to an asynchronous task executor object;
after all task instructions are executed, the recorded task results are put into a result queue, and the main control end acquires the task results from the result queue and stores the task results into a database;
the execution instruction unit comprises a data collection subunit and is used for the task executor to receive a data collection task, and after parameters are extracted, the nodes send the data collection task to a space search engine according to parameter configuration to obtain matching data; after the data collection task is completed, the main control end puts the result of the data collection task into the task queue again to be used as the input of the vulnerability detection task;
the system also comprises a host detection subunit, which is used for the node to acquire the task from the task queue again and then execute the host detection action: the task executor receives the vulnerability exploiting task and transmits a data collection result as a unique data source of the vulnerability exploitation;
the execution instruction unit also comprises an application identification subunit, which is used for calling a built-in application identification algorithm of the task executor and carrying out application identification by combining an application identification fingerprint transmitted by the structural body;
the execution instruction unit also comprises a vulnerability verification subunit, which is used for setting a vulnerability verification PoC frame in the node when the application identification is successful, the node sends a request with a carrier to the target website, and the verification state is judged according to the target response;
the execution instruction unit also comprises a vulnerability exploiting subunit, which is used for calling a vulnerability exploiting algorithm of the task executor when vulnerability verification is successful, and the node initiates a request to the target website and judges whether vulnerability exploitation is successful according to the target response;
the node destruction unit is used for finishing the self process of the node and deleting the self configuration for self destruction when the main control end issues the node destruction instruction.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210139075.2A CN114489995B (en) | 2022-02-15 | 2022-02-15 | Distributed scheduling processing method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210139075.2A CN114489995B (en) | 2022-02-15 | 2022-02-15 | Distributed scheduling processing method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114489995A true CN114489995A (en) | 2022-05-13 |
| CN114489995B CN114489995B (en) | 2022-09-30 |
Family
ID=81480024
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210139075.2A Active CN114489995B (en) | 2022-02-15 | 2022-02-15 | Distributed scheduling processing method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114489995B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115361455A (en) * | 2022-08-22 | 2022-11-18 | 中能融合智慧科技有限公司 | Data transmission and storage method and device and computer equipment |
Citations (19)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1655557A (en) * | 2005-02-25 | 2005-08-17 | 上海高智软件系统有限公司 | Communication equipment for home position register |
| CN1787438A (en) * | 2005-11-30 | 2006-06-14 | 浙江大学 | Electronic business network system and intelligent information terminal thereof |
| CN101464922A (en) * | 2009-01-22 | 2009-06-24 | 中国人民解放军国防科学技术大学 | Computer architecture scheme parallel simulation optimization method based on cluster system |
| CN101558599A (en) * | 2006-05-30 | 2009-10-14 | 小川惠子 | Client device, mail system, program, and recording medium |
| CN101694626A (en) * | 2009-09-30 | 2010-04-14 | 曙光信息产业(北京)有限公司 | Script execution system and method |
| US8843622B1 (en) * | 2011-12-19 | 2014-09-23 | Cisco Technology, Inc. | System and method to contact and maintain status of managed devices |
| CN104951474A (en) * | 2014-03-31 | 2015-09-30 | 阿里巴巴集团控股有限公司 | Method and device for acquiring MySQL binlog incremental logs |
| CN106101213A (en) * | 2016-06-08 | 2016-11-09 | 四川新环佳科技发展有限公司 | Information-distribution type storage method |
| CN106302660A (en) * | 2016-08-01 | 2017-01-04 | 成都鼎智汇科技有限公司 | Medicine information distributed storage method |
| CN107302475A (en) * | 2017-07-06 | 2017-10-27 | 郑州云海信息技术有限公司 | The method of testing and device of a kind of Based on Distributed storage cluster |
| CN108012305A (en) * | 2017-12-19 | 2018-05-08 | 欧普照明股份有限公司 | Smart machine matches somebody with somebody network method, system and smart machine |
| CN108038013A (en) * | 2017-11-30 | 2018-05-15 | 海尔优家智能科技（北京）有限公司 | Distributed performance test method and device and computer-readable recording medium |
| WO2018127045A1 (en) * | 2017-01-09 | 2018-07-12 | 中国移动通信有限公司研究院 | Method and device for recognizing and processing failure of base station |
| CN108280023A (en) * | 2017-01-04 | 2018-07-13 | 中兴通讯股份有限公司 | Task executing method, device and server |
| CN207766560U (en) * | 2017-12-19 | 2018-08-24 | 欧普照明股份有限公司 | Smart machine and smart machine distribution network systems |
| CN111181894A (en) * | 2018-11-09 | 2020-05-19 | 北京天德科技有限公司 | Efficient and safe protocol for block link point communication |
| US20210034398A1 (en) * | 2019-07-31 | 2021-02-04 | Rubrik, Inc. | Streaming database cloning using cluster live mounts |
| CN112632566A (en) * | 2021-03-05 | 2021-04-09 | 腾讯科技（深圳）有限公司 | Vulnerability scanning method and device, storage medium and electronic equipment |
| US20210211450A1 (en) * | 2020-01-02 | 2021-07-08 | Saudi Arabian Oil Company | Method and system for prioritizing and remediating security vulnerabilities based on adaptive scoring |
-
2022
- 2022-02-15 CN CN202210139075.2A patent/CN114489995B/en active Active
Patent Citations (19)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1655557A (en) * | 2005-02-25 | 2005-08-17 | 上海高智软件系统有限公司 | Communication equipment for home position register |
| CN1787438A (en) * | 2005-11-30 | 2006-06-14 | 浙江大学 | Electronic business network system and intelligent information terminal thereof |
| CN101558599A (en) * | 2006-05-30 | 2009-10-14 | 小川惠子 | Client device, mail system, program, and recording medium |
| CN101464922A (en) * | 2009-01-22 | 2009-06-24 | 中国人民解放军国防科学技术大学 | Computer architecture scheme parallel simulation optimization method based on cluster system |
| CN101694626A (en) * | 2009-09-30 | 2010-04-14 | 曙光信息产业(北京)有限公司 | Script execution system and method |
| US8843622B1 (en) * | 2011-12-19 | 2014-09-23 | Cisco Technology, Inc. | System and method to contact and maintain status of managed devices |
| CN104951474A (en) * | 2014-03-31 | 2015-09-30 | 阿里巴巴集团控股有限公司 | Method and device for acquiring MySQL binlog incremental logs |
| CN106101213A (en) * | 2016-06-08 | 2016-11-09 | 四川新环佳科技发展有限公司 | Information-distribution type storage method |
| CN106302660A (en) * | 2016-08-01 | 2017-01-04 | 成都鼎智汇科技有限公司 | Medicine information distributed storage method |
| CN108280023A (en) * | 2017-01-04 | 2018-07-13 | 中兴通讯股份有限公司 | Task executing method, device and server |
| WO2018127045A1 (en) * | 2017-01-09 | 2018-07-12 | 中国移动通信有限公司研究院 | Method and device for recognizing and processing failure of base station |
| CN107302475A (en) * | 2017-07-06 | 2017-10-27 | 郑州云海信息技术有限公司 | The method of testing and device of a kind of Based on Distributed storage cluster |
| CN108038013A (en) * | 2017-11-30 | 2018-05-15 | 海尔优家智能科技（北京）有限公司 | Distributed performance test method and device and computer-readable recording medium |
| CN108012305A (en) * | 2017-12-19 | 2018-05-08 | 欧普照明股份有限公司 | Smart machine matches somebody with somebody network method, system and smart machine |
| CN207766560U (en) * | 2017-12-19 | 2018-08-24 | 欧普照明股份有限公司 | Smart machine and smart machine distribution network systems |
| CN111181894A (en) * | 2018-11-09 | 2020-05-19 | 北京天德科技有限公司 | Efficient and safe protocol for block link point communication |
| US20210034398A1 (en) * | 2019-07-31 | 2021-02-04 | Rubrik, Inc. | Streaming database cloning using cluster live mounts |
| US20210211450A1 (en) * | 2020-01-02 | 2021-07-08 | Saudi Arabian Oil Company | Method and system for prioritizing and remediating security vulnerabilities based on adaptive scoring |
| CN112632566A (en) * | 2021-03-05 | 2021-04-09 | 腾讯科技（深圳）有限公司 | Vulnerability scanning method and device, storage medium and electronic equipment |
Non-Patent Citations (3)
| Title |
|---|
| CHANDRADEO KUMAR RAJAK ET AL.: "《Real-time web based Timing display Application for Test Range Applications》", 《2021 2ND INTERNATIONAL CONFERENCE ON RANGE TECHNOLOGY (ICORT)》 * |
| 许子先: "云计算环境下远程安全评估技术研究", 《网络安全技术与应用》 * |
| 高健: "《基于网络漏洞分析的安全设备部署设计研究》", 《网络安全技术与应用》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115361455A (en) * | 2022-08-22 | 2022-11-18 | 中能融合智慧科技有限公司 | Data transmission and storage method and device and computer equipment |
| CN115361455B (en) * | 2022-08-22 | 2024-01-23 | 中能融合智慧科技有限公司 | Data transmission storage method and device and computer equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114489995B (en) | 2022-09-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110297689B (en) | Intelligent contract execution method, device, equipment and medium | |
| EP3178011B1 (en) | Method and system for facilitating terminal identifiers | |
| CN112256558B (en) | Test case generation method and device, computer equipment and storage medium | |
| CN101777062A (en) | context-aware real-time computer-protection systems and methods | |
| CN113570466B (en) | Transaction data processing method and device and readable storage medium | |
| CN109995523B (en) | Activation code management method and device and activation code generation method and device | |
| CN111885128A (en) | Identity management method based on block chain | |
| US20080027866A1 (en) | System and method for authenticating file content | |
| WO2020224100A1 (en) | Blockchain configuration file processing apparatus, system, and method, and storage medium | |
| CN110210845B (en) | Method, apparatus, medium, and computing device for blockchain data migration | |
| US20210174331A1 (en) | Transferring a customer from an atm transaction to a device-based transaction during an error state, and applications thereof | |
| CN114489995B (en) | Distributed scheduling processing method and system | |
| CN113315828A (en) | Traffic recording method and device, traffic recording equipment and storage medium | |
| CN112200680B (en) | Block link point management method, device, computer and readable storage medium | |
| CN112069529B (en) | Block chain-based volume management method and device, computer and storage medium | |
| CN112583805A (en) | Data processing method and device based on block chain, electronic equipment and storage medium | |
| CN116743619A (en) | Network service testing method, device, equipment and storage medium | |
| CN115665265B (en) | Request processing method, device, equipment, storage medium and system | |
| CN111722994A (en) | Task request response method and device | |
| CN111666132A (en) | Distributed transaction implementation method, device, computer system and readable storage medium | |
| JP2002366528A (en) | Security system for personal identification | |
| CN115203063B (en) | Playback method and system of production flow re-running risk program based on real-time recording | |
| CN111722946A (en) | Distributed transaction processing method and device, computer equipment and readable storage medium | |
| JP2001147888A (en) | Method for recognizing connecting device | |
| CN113971275A (en) | Event parallel computing method and device based on server development |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP01 | Change in the name or title of a patent holder | ||
| CP01 | Change in the name or title of a patent holder |
Address after: 100094 building 6, yard 9, FengHao East Road, Haidian District, Beijing Patentee after: Yongxin Zhicheng Technology Group Co.,Ltd. Address before: 100094 building 6, yard 9, FengHao East Road, Haidian District, Beijing Patentee before: BEIJING YONGXIN ZHICHENG TECHNOLOGY CO.,LTD. |