CN110750497A - Data scheduling system - Google Patents
Data scheduling system Download PDFInfo
- Publication number
- CN110750497A CN110750497A CN201911037777.4A CN201911037777A CN110750497A CN 110750497 A CN110750497 A CN 110750497A CN 201911037777 A CN201911037777 A CN 201911037777A CN 110750497 A CN110750497 A CN 110750497A
- Authority
- CN
- China
- Prior art keywords
- name
- user
- data
- container
- analyzed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/113—Details of archiving
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/119—Details of migration of file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/122—File system administration, e.g. details of archiving or snapshots using management policies
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/14—Session management
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/14—Session management
- H04L67/143—Termination or inactivation of sessions, e.g. event-controlled end of session
- H04L67/145—Termination or inactivation of sessions, e.g. event-controlled end of session avoiding end of session, e.g. keep-alive, heartbeats, resumption message or wake-up for inactive or interrupted session
Abstract
The invention provides a data scheduling system, which discovers the operation condition of a user on an object in a storage server and the operated object by monitoring and analyzing a network message through a network, records the discovered operation related to the change of the object into a database, and only stores the state of the corresponding operated object in the database. The user can configure the management strategy for the data, and the task scheduling module can schedule the tasks according to the management strategy configured by the user. For the tasks with the blue-ray storage as the target address, the task scheduling module scans the database, and selects the object information which is in the storage server and accords with the configuration strategy and the position of the available blue-ray storage as the target storage position through the blue-ray storage node management module. The method can be widely applied to the storage cluster supporting the standard object storage protocol, realizes the real-time discovery of object changes, automatically schedules data, does not bring extra pressure to storage, and does not influence the normal operation and bandwidth of services.
Description
Technical Field
The invention relates to a system for automatically archiving and retrieving object data, in particular to a data scheduling system.
Background
With the development of society, human society generates a large amount of data every day, and a large amount of data needs to be permanently stored for a long time, but the current way of storing data by a magnetic storage cluster has the problems of high cost and high power consumption. In order to solve this problem, it is necessary to store data that is not frequently used by low-cost and low-power consumption devices.
With current technology, blu-ray disc storage is one option for low cost long term storage. The data is initially stored in the magnetic storage, which is a rule used by people, and as to how the data stored in the magnetic storage can be stored in the optical disc, the industry has some solutions, some solutions are lack of function (only supporting manual writing and retrieving of files into and from the optical disc), some solutions are too high in coupling degree (only using storage software of the manufacturer or relying on storage of a specific manufacturer), or too much pressure is applied to storage.
Disclosure of Invention
In order to solve the above problem, the present invention provides a data scheduling system.
In order to realize the purpose, the invention adopts the technical scheme that:
a data scheduling system comprises a control center, an Agent module, a storage server and a blue-ray storage, wherein the control center comprises a data management strategy module, a blue-ray storage node management module, a task scheduling module, a monitoring and analyzing module and a database;
the data management strategy module is used for finishing the strategy configuration of a container by a user, and comprises an enabling strategy, a configuring strategy, a deleting strategy, a modifying strategy and a disabling strategy;
the blue-ray storage node management module is used for providing target storage position information of blue-ray storage for the task scheduling module, and the target storage position information comprises a Uniform Resource Locator (URL) and authentication information;
the monitoring and analyzing module is used for monitoring the network flow of the storage server, analyzing the operation information of the user on the data in the storage server from the network flow, acquiring the data change condition information from the operation information and recording the data change condition information into the database;
the task scheduling module is used for scanning the database regularly, judging whether a container in the storage server meets a data management strategy configured by a user according to the data change condition information acquired by the monitoring and analyzing module, and if so, generating a data scheduling task and sending the data scheduling task to a message queue;
the Agent module is used for executing data scheduling and finishing backing up data from the storage server to the blue-ray storage or retrieving the data from the blue-ray storage to the storage server according to tasks in the message queue;
the database is: the data management system is used for storing monitored data operation information of the data in the storage server and a data management strategy configured by a user.
Preferably, the monitoring and analyzing module monitors an IP address and a network port of the storage server in a bypass monitoring manner, when a user operates data in the storage server through a client or writes data into the storage server, the monitoring and analyzing module captures and filters a data packet through a configured filter, acquires an http/http packet for operating data in the storage server or writing data into the storage server, then performs packet analysis, needs to analyze a request and response part of the http/http packet, acquires a request method, corresponding user and container information and a processing result from the request and response part, and stores/updates information acquired after analyzing the http/http packet into a database.
Preferably, the http/https packet includes a request line and a request header, and the request line includes a request method field: PUT/GET/HEAD/POST/DELETE/COPY, wherein the request header comprises a Content-Length field and a Destination field;
after the PUT method is analyzed, the user uploads an object or a container to a storage server at a client side, the URL field needs to be analyzed continuously, the URL field sequentially comprises a user name, a container name and an object name, the URL field is analyzed sequentially according to the sequence, if only the user name is analyzed, the container name is not analyzed subsequently, the operation of creating an account is explained, and the analysis is abandoned continuously; if the container name and the object name are resolved or only the container name is resolved, the Content-Length field in the request header needs to be resolved continuously so as to obtain the size of the object; then, capturing a response message aiming at the request, analyzing a state code from a response line, judging that the user uploads successfully if the state code is 200-;
after the analysis is carried out on the GET method, the user needs to read data from the storage server at the client, the GET method cannot modify any container in the storage server, and the http/https message analysis is finished;
after the analysis of the HEAD method, the user is shown to check the metadata information corresponding to the user, the container or the object in the storage server at the client, the HEAD method does not modify the container in the storage server, and the http/https message analysis is finished;
after the POST method is analyzed, the fact that a client needs to create, update or modify metadata information corresponding to a user, a container or an object is described, a URL field behind the user, the URL field sequentially comprises a user name, a container name and an object name, the URL field is sequentially analyzed according to the sequence, if only the user name is analyzed, or only the container name is analyzed, or the user name and the container name are analyzed, or the user name, the container name and the object name are analyzed, a response message corresponding to the request needs to be captured, a state code is analyzed from a response line, if the state code is 200-value 299, the user creation, update or modification success is judged, then a response head is analyzed, the current modification time is obtained from a Date field, and finally the metadata information corresponding to the analyzed user, container or object and the current modification time information are stored/updated in a database;
after the DELETE method is analyzed, the client needs to DELETE an object or a container from the storage server, the URL field after the deletion needs to be continuously analyzed, the URL field sequentially comprises a user name, a container name and an object name, the URL field is sequentially analyzed according to the sequence, if only the container name or the object name is analyzed, or the container name and the object name are analyzed, a response message aiming at the request needs to be captured, a state code is analyzed from a response line, if the state code is 200-plus 299, the user is judged to successfully DELETE the object or the container, and finally information corresponding to the container name and the object name of the user is deleted from a database;
after the COPY method is analyzed, the fact that a client needs to COPY an object is indicated, a URL field behind the client needs to be continuously analyzed, the URL field sequentially comprises a user name, a container name and an object name, the object name is sequentially analyzed according to the sequence, then a Destination field in a request header is analyzed so as to record a target position and a name, then a response message aiming at the request is captured, a state code is analyzed from a response line, if the state code is 200-.
Furthermore, the configuration policy of the data management policy module is configured by setting a time threshold of an archiving policy for a container by a user, the policy can be started in units of days for the configured policy, after the policy takes effect, the task scheduling module acquires the modification time of an object from the database and also acquires the time of the current system, a time difference is obtained by using the two times, and finally the time difference is compared with the configured time threshold, and the object with the time difference exceeding the set time threshold needs to be archived to a blue-ray storage; the modification strategy is to modify the time threshold of the configuration strategy; deleting a policy refers to deleting the configuration policy; disabling a policy refers to ceasing to use a configured configuration policy, rendering it non-functional.
Furthermore, the task scheduling module generates a data scheduling task which comprises a URL, a project name, a user name and a password of a container to be archived, wherein the container to be archived is located in the storage server; the URL, the project name, the user name and the password which are positioned at the target position of the blue-ray storage are set when a user creates a new container in the blue-ray storage through a client or are acquired when the user associates an existing container, and are stored in a blue-ray storage node management module.
Furthermore, the Agent module is deployed on the storage server, and can send a heartbeat message to the task scheduling module of the control center in units of seconds, the task scheduling module receives the heartbeat message sent by the Agent module, namely, the Agent module is considered to be normal, if the heartbeat message is not received for more than 3 times, the Agent module is considered to be in fault, and the task scheduling module cannot send a task to the message queue.
Further, the method for retrieving the Agent module from the blue-ray storage to the storage server is as follows: the method comprises the steps that a user initiates a read request from the blue-ray storage at a client, the information of the read request comprises a URL (uniform resource locator) of an object to be read, a project name for accessing the blue-ray storage, a user name and a password, a task scheduling module sends a read request message to a message queue, an Agent module acquires a task of the read request from the message queue, then the Agent module initiates the read request to the blue-ray storage, and the read object data is written into a storage server.
Further, the database type is a MongoDB database.
The method is convenient to use, low in coupling degree, capable of being widely applied to the storage cluster supporting the standard object storage protocol, capable of finding object changes in real time and automatically scheduling data, and does not bring extra pressure to storage, and does not influence normal operation of business and bandwidth of business.
Drawings
The accompanying drawings are included to provide a further understanding of the invention.
In the drawings:
fig. 1 is a system diagram of a data scheduling system according to the present invention.
Fig. 2 is a block diagram of a workflow of a method for parsing to PUT by a monitoring parsing module according to the present invention.
Fig. 3 is a block diagram of a workflow of a method for analyzing to GET or HEAD by the monitoring and analyzing module according to the present invention.
FIG. 4 is a block diagram of a workflow of a method for parsing a POST by a monitoring and parsing module according to the present invention.
FIG. 5 is a block diagram of the workflow of the method for parsing to DELETE by the interception parsing module according to the present invention.
Fig. 6 is a flowchart of the operation of the method for analyzing COPY by the monitoring and analyzing module according to the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
As shown in fig. 1, a data scheduling system includes a control center, an Agent module, a storage server, and a blue-ray storage, where the control center includes a data management policy module, a blue-ray storage node management module, a task scheduling module, a monitoring and parsing module, and a database;
the data management strategy module is used for finishing the strategy configuration of a container by a user, and comprises an enabling strategy, a configuring strategy, a deleting strategy, a modifying strategy and a disabling strategy;
the blue-ray storage node management module is used for providing target storage position information of blue-ray storage for the task scheduling module, and the target storage position information comprises a Uniform Resource Locator (URL) and authentication information;
the monitoring and analyzing module is used for monitoring the network flow of the storage server, analyzing the operation information of the user on the data in the storage server from the network flow, acquiring the data change condition information from the operation information and recording the data change condition information into the database;
the task scheduling module is used for scanning the database regularly, judging whether a container in the storage server meets a data management strategy configured by a user according to the data change condition information acquired by the monitoring and analyzing module, and if so, generating a data scheduling task and sending the data scheduling task to a message queue;
the Agent module is used for executing data scheduling and finishing backing up data from the storage server to the blue-ray storage or retrieving the data from the blue-ray storage to the storage server according to tasks in the message queue;
the database is: the data management system is used for storing monitored data operation information of data in the storage server and a data management strategy configured by a user, and the type of the database is a MongoDB database.
The monitoring and analyzing module monitors an IP address and a network port of the storage server in a bypass monitoring mode, when a user operates data in the storage server or writes data into the storage server through a client or other software, a configured filter is used for capturing and filtering a data packet, an http/http message for operating the data in the storage server or writing the data into the storage server is obtained, then message analysis is carried out, a request and response part of the http/http message needs to be analyzed, a request method, corresponding user and container information and a processing result are obtained from the request and response part, and the information obtained after the http/http message is analyzed is stored/updated in a database. Message parsing is performed by using a swift template.
The http/https message comprises a request line and a request header, wherein the request line comprises a request method field: PUT/GET/HEAD/POST/DELETE/COPY, wherein the request header comprises a Content-Length field and a Destination field;
after the PUT method is analyzed, the user uploads an object or a container to a storage server at a client side, the URL field needs to be analyzed continuously, the URL field sequentially comprises a user name (account), a container name (container) and an object name (object), the URL field is analyzed sequentially according to the sequence, if only the user name is analyzed, the container name is not analyzed subsequently, the operation of creating an account is described, and the analysis is abandoned continuously; if the container name and the object name are resolved or only the container name is resolved, the Content-Length field in the request header needs to be resolved continuously so as to obtain the size of the object; then, capturing a response message aiming at the request, analyzing a state code from a response line, judging that the user uploads successfully if the state code is 200-;
after the analysis is carried out on the GET method, the user needs to read data from the storage server at the client, the GET method cannot modify any container in the storage server, and the http/https message analysis is finished;
after the analysis of the HEAD method, the user is shown to check the metadata information corresponding to the user, the container or the object in the storage server at the client, the HEAD method does not modify the container in the storage server, and the http/https message analysis is finished;
after the POST method is analyzed, the fact that a client needs to create, update or modify metadata information corresponding to a user, a container or an object is described, a URL field behind the user, the URL field sequentially comprises a user name, a container name and an object name, the URL field is sequentially analyzed according to the sequence, if only the user name is analyzed, or only the container name is analyzed, or the user name and the container name are analyzed, or the user name, the container name and the object name are analyzed, a response message corresponding to the request needs to be captured, a state code is analyzed from a response line, if the state code is 200-value 299, the user creation, update or modification success is judged, then a response head is analyzed, the current modification time is obtained from a Date field, and finally the metadata information corresponding to the analyzed user, container or object and the current modification time information are stored/updated in a database;
after the DELETE method is analyzed, the client needs to DELETE an object or a container from the storage server, the URL field after the deletion needs to be continuously analyzed, the URL field sequentially comprises a user name, a container name and an object name, the URL field is sequentially analyzed according to the sequence, if only the container name or the object name is analyzed, or the container name and the object name are analyzed, a response message aiming at the request needs to be captured, a state code is analyzed from a response line, if the state code is 200-plus 299, the user is judged to successfully DELETE the object or the container, and finally information corresponding to the container name and the object name of the user is deleted from a database;
after the COPY method is analyzed, the fact that a client needs to COPY an object is indicated, a URL field behind the client needs to be continuously analyzed, the URL field sequentially comprises a user name, a container name and an object name, the object name is sequentially analyzed according to the sequence, then a Destination field in a request header is analyzed so as to record a target position and a name, then a response message aiming at the request is captured, a state code is analyzed from a response line, if the state code is 200-.
The configuration strategy of the data management strategy module is configured by setting a time threshold of a filing strategy for a container by a user, a plurality of strategies can be simultaneously configured by taking days as a unit, the strategy can be started for the configured strategy, after the strategy takes effect, the task scheduling module can acquire the modification time of an object from the database and the time of the current system at the same time, a time difference is obtained by using the two times, and finally the time difference is compared with the configured time threshold, and the object of which the time difference exceeds the set time threshold needs to be filed on a blue-ray storage; the modification strategy is to modify the time threshold of the configuration strategy; deleting a policy refers to deleting the configuration policy; disabling a policy refers to ceasing to use a configured configuration policy, rendering it non-functional. The configured filing strategy is configured according to the use scene of a user, and in the medical industry, after a patient visits for three months, an electronic medical record, an inspection report and the like need to be filed; in the public security department, the collected fingerprint information, the image identification information and the video information are placed in different containers, the fingerprint information is required to be filed in one week after collection, the image identification information is required to be filed in one month after collection, the video information is required to be filed in three months after collection, and the like.
The task scheduling module generates a data scheduling task which comprises a URL, a project name, a user name and a password of a container to be filed in a storage server, and also comprises a URL, a project name, a user name and a password of a target storage position in blue-ray storage, wherein the URL, the project name, the user name and the password of the container to be filed in the storage server are obtained by scanning a database; the URL, the project name, the user name and the password which are positioned at the target position of the blue-ray storage are set when a user creates a new container in the blue-ray storage through a client or are acquired when the user associates an existing container, and are stored in a blue-ray storage node management module.
The Agent module is deployed on a storage server or a control center, and can send heartbeat messages to a task scheduling module of the control center in units of seconds, the task scheduling module receives the heartbeat messages sent by the Agent module, namely the Agent module is considered to be normal, if the heartbeat messages are not received for more than 3 times, the Agent module is considered to be in fault, and the task scheduling module cannot send tasks to a message queue. The Agent module does not actively acquire the running state of the task scheduling module in the control center, after the task scheduling module finishes issuing the task, the task is stored in the message queue, the Agent module takes out the task from the message queue and executes the task, at the moment, the execution of the task is not influenced even if the control center fails, because the Agent module executes the task without depending on the control center, and data transmission does not need to pass through the control center, the single-point failure of the control center can be avoided, the actual read-write speed of the system for data scheduling is the sum of the read-write speed of the node where each Agent module is located, but not the read-write speed of a single server, namely 2 times of the read-write speed of the single server, and the transmission efficiency of the data scheduling is greatly improved.
The method for retrieving the Agent module from the blue-ray storage to the storage server comprises the following steps: the method comprises the steps that a user initiates a read request from the blue-ray storage at a client, the information of the read request comprises a URL (uniform resource locator) of an object to be read, a project name for accessing the blue-ray storage, a user name and a password, a task scheduling module sends a read request message to a message queue, an Agent module acquires a task of the read request from the message queue, then the Agent module initiates the read request to the blue-ray storage, and the read object data is written into a storage server.
The data scheduling system monitors through the network and analyzes the network message to find the operation condition of the user on the object in the storage server and the operation object, records the found operation related to the object change into the database, and only stores the state of the corresponding operation object in the database. A user can configure a management strategy for data through a client, and a task scheduling module can schedule tasks according to the management strategy configured by the user. For the tasks with the blue-ray storage as the target address, the task scheduling module scans the database, and selects the object information which is in the storage server and accords with the configuration strategy and the position of the available blue-ray storage as the target storage position through the blue-ray storage node management module.
The task scheduling module of the control center scans own database according to the filing strategy configured by the user, and executes filing operation aiming at the object which accords with the filing strategy:
1. the task scheduling module initiates an archiving request to the Agent module through A Message Queue (AMQP), wherein the request contains an object URL (uniform resource locator), a project name (project name), a user name (user name), a password (password) or a key pair (access key, secret key) which are acquired from a database and need to be archived, and also contains the URL, the project name (project name), the user name (user name), the password (password) or the key pair (access key, secret key) of an archived target position acquired from the blue-ray storage node management module.
And 2, the Agent module acquires the filing request from the message queue, and searches and reads the object to be filed from the storage server according to the request content sent by the task scheduling module.
And 3, the Agent module is connected with the blue-ray storage according to the target position information of the blue-ray storage designated by the blue-ray storage node management module of the control center, and the read object data is written into the blue-ray storage through the swift interface of the blue-ray storage.
The user can configure the strategy according to the own requirement and can also manually schedule the object, thereby realizing the purposes of reasonably utilizing the magnetic storage space, permanently storing the object data and meeting the access requirement of the user.
The customer reads the data from the blue storage through the control center:
1. the client initiates a read request from the blue-ray storage through a task scheduling module of the control center, and the task scheduling module sends the read request message to a message queue.
And 2, the Agent module acquires relevant information read from the Blu-ray storage from the message queue (including URL, project name, user name, password or key pair of the Blu-ray storage object required to be read from the Blu-ray storage node management module).
And 3, the Agent module initiates a reading request to the blue-ray storage according to the information of the acquired message queue, and then writes the read object data into the storage server.
Although the present invention has been described in detail with reference to the foregoing embodiments, it will be apparent to those skilled in the art that various changes in the embodiments and/or modifications of the invention can be made, and equivalents and modifications of some features of the invention can be made without departing from the spirit and scope of the invention.
Claims (8)
1. A data scheduling system, characterized by: the system comprises a control center, an Agent module, a storage server and a blue-ray storage, wherein the control center comprises a data management strategy module, a blue-ray storage node management module, a task scheduling module, a monitoring and analyzing module and a database;
the data management strategy module is used for finishing the strategy configuration of a container by a user, and comprises an enabling strategy, a configuring strategy, a deleting strategy, a modifying strategy and a disabling strategy;
the blue-ray storage node management module is used for providing target storage position information of blue-ray storage for the task scheduling module, and the target storage position information comprises a Uniform Resource Locator (URL) and authentication information;
the monitoring and analyzing module is used for monitoring the network flow of the storage server, analyzing the operation information of the user on the data in the storage server from the network flow, acquiring the data change condition information from the operation information and recording the data change condition information into the database;
the task scheduling module is used for scanning the database regularly, judging whether a container in the storage server meets a data management strategy configured by a user according to the data change condition information acquired by the monitoring and analyzing module, and if so, generating a data scheduling task and sending the data scheduling task to a message queue;
the Agent module is used for executing data scheduling and finishing backing up data from the storage server to the blue-ray storage or retrieving the data from the blue-ray storage to the storage server according to tasks in the message queue;
the database is: the data management system is used for storing monitored data operation information of the data in the storage server and a data management strategy configured by a user.
2.A data scheduling system according to claim 1, wherein: the monitoring and analyzing module monitors an IP address and a network port of the storage server in a bypass monitoring mode, when a user operates data in the storage server through a client or writes data into the storage server, a configured filter captures and filters a data packet, an http/https message for operating the data in the storage server or writing the data into the storage server is obtained, then message analysis is carried out, a request and response part of the http/https message needs to be analyzed, a request method, corresponding user and container information are obtained from the request and response part, and the information obtained after the http/https message is analyzed is stored/updated in a database.
3.A data scheduling system according to claim 2, wherein: the http/https message comprises a request line and a request header, wherein the request line comprises a request method field: PUT/GET/HEAD/POST/DELETE/COPY, wherein the request header comprises a Content-Length field and a Destination field;
after the PUT method is analyzed, the user uploads an object or a container to a storage server at a client side, the URL field needs to be analyzed continuously, the URL field sequentially comprises a user name, a container name and an object name, the URL field is analyzed sequentially according to the sequence, if only the user name is analyzed, the container name is not analyzed subsequently, the operation of creating an account is explained, and the analysis is abandoned continuously; if the container name and the object name are resolved or only the container name is resolved, the Content-Length field in the request header needs to be resolved continuously so as to obtain the size of the object; then, capturing a response message aiming at the request, analyzing a state code from a response line, judging that the user uploads successfully if the state code is 200-;
after the analysis is carried out on the GET method, the user needs to read data from the storage server at the client, the GET method cannot modify any container in the storage server, and the http/https message analysis is finished;
after the analysis of the HEAD method, the user is shown to check the metadata information corresponding to the user, the container or the object in the storage server at the client, the HEAD method does not modify the container in the storage server, and the http/https message analysis is finished;
after the POST method is analyzed, the fact that a client needs to create, update or modify metadata information corresponding to a user, a container or an object is described, a URL field behind the user, the URL field sequentially comprises a user name, a container name and an object name, the URL field is sequentially analyzed according to the sequence, if only the user name is analyzed, or only the container name is analyzed, or the user name and the container name are analyzed, or the user name, the container name and the object name are analyzed, a response message corresponding to the request needs to be captured, a state code is analyzed from a response line, if the state code is 200-value 299, the user creation, update or modification success is judged, then a response head is analyzed, the current modification time is obtained from a Date field, and finally the metadata information corresponding to the analyzed user, container or object and the current modification time information are stored/updated in a database;
after the DELETE method is analyzed, the client needs to DELETE an object or a container from the storage server, the URL field after the deletion needs to be continuously analyzed, the URL field sequentially comprises a user name, a container name and an object name, the URL field is sequentially analyzed according to the sequence, if only the container name or the object name is analyzed, or the container name and the object name are analyzed, a response message aiming at the request needs to be captured, a state code is analyzed from a response line, if the state code is 200-plus 299, the user is judged to successfully DELETE the object or the container, and finally information corresponding to the container name and the object name of the user is deleted from a database;
after the COPY method is analyzed, the fact that a client needs to COPY an object is indicated, a URL field behind the client needs to be continuously analyzed, the URL field sequentially comprises a user name, a container name and an object name, the object name is sequentially analyzed according to the sequence, then a Destination field in a request header is analyzed so as to record a target position and a name, then a response message aiming at the request is captured, a state code is analyzed from a response line, if the state code is 200-.
4. A data scheduling system according to claim 3 wherein: the configuration strategy of the data management strategy module is configured by setting a time threshold of an archiving strategy for a container by a user, the strategy can be started by taking days as a unit for the configured strategy, after the strategy takes effect, the task scheduling module can acquire the modification time of an object from the database and the time of the current system at the same time, a time difference is obtained by using the two times, and finally the time difference is compared with the configured time threshold, and the object of which the time difference exceeds the set time threshold needs to be archived to a blue-ray storage; the modification strategy is to modify the time threshold of the configuration strategy; deleting a policy refers to deleting the configuration policy; disabling a policy refers to ceasing to use a configured configuration policy, rendering it non-functional.
5. A data scheduling system according to claim 3 wherein: the task scheduling module generates a data scheduling task which comprises a URL, a project name, a user name and a password of a container to be filed in a storage server, and also comprises a URL, a project name, a user name and a password of a target storage position in blue-ray storage, wherein the URL, the project name, the user name and the password of the container to be filed in the storage server are obtained by scanning a database; the URL, the project name, the user name and the password which are positioned at the target position of the blue-ray storage are set when a user creates a new container in the blue-ray storage through a client or are acquired when the user associates an existing container, and are stored in a blue-ray storage node management module.
6. A data scheduling system according to claim 4 wherein: the Agent module is deployed on the storage server and can send heartbeat messages to the task scheduling module of the control center in units of seconds, the task scheduling module receives the heartbeat messages sent by the Agent module, namely the Agent module is considered to be normal, if the heartbeat messages are not received for more than 3 times, the Agent module is considered to be in fault, and the task scheduling module cannot send tasks to the message queue.
7. A data scheduling system according to claim 5 wherein: the method for retrieving the Agent module from the blue-ray storage to the storage server comprises the following steps: the method comprises the steps that a user initiates a read request from the blue-ray storage at a client, the information of the read request comprises a URL (uniform resource locator) of an object to be read, a project name for accessing the blue-ray storage, a user name and a password, a task scheduling module sends a read request message to a message queue, an Agent module acquires a task of the read request from the message queue, then the Agent module initiates the read request to the blue-ray storage, and the read object data is written into a storage server.
8. A data scheduling system according to claim 1, wherein: the database type is a MongoDB database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911037777.4A CN110750497B (en) | 2019-10-29 | 2019-10-29 | Data scheduling system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911037777.4A CN110750497B (en) | 2019-10-29 | 2019-10-29 | Data scheduling system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110750497A true CN110750497A (en) | 2020-02-04 |
| CN110750497B CN110750497B (en) | 2023-09-26 |
Family
ID=69280770
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911037777.4A Active CN110750497B (en) | 2019-10-29 | 2019-10-29 | Data scheduling system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110750497B (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111736956A (en) * | 2020-06-29 | 2020-10-02 | 苏州浪潮智能科技有限公司 | Container service deployment method, device, equipment and readable storage medium |
| CN111934723A (en) * | 2020-08-26 | 2020-11-13 | 上海仪电（集团）有限公司中央研究院 | Bypass interception Bluetooth communication device, method and application thereof |
| CN112818059A (en) * | 2021-01-27 | 2021-05-18 | 百果园技术(新加坡)有限公司 | Information real-time synchronization method and device based on container release platform |
| CN113032598A (en) * | 2021-04-12 | 2021-06-25 | 郑州航空工业管理学院 | Image design visual transmission system based on big data |
| CN115242677A (en) * | 2021-04-23 | 2022-10-25 | 中国移动通信集团四川有限公司 | Home-wide user state monitoring system, method and device |
| CN117076094A (en) * | 2023-10-16 | 2023-11-17 | 中国船舶集团有限公司第七〇七研究所 | Method for concurrently processing multiple tasks of cryptographic operation |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160019254A1 (en) * | 2014-07-16 | 2016-01-21 | Facebook. Inc. | Tiered data storage architecture |
| CN106648961A (en) * | 2016-09-27 | 2017-05-10 | 上海爱数信息技术股份有限公司 | Integrated blue-ray disc jukebox backup and archiving method |
| CN106649467A (en) * | 2016-09-27 | 2017-05-10 | 上海爱数信息技术股份有限公司 | Blue-ray disc jukebox archiving management method and system |
| WO2018145739A1 (en) * | 2017-02-08 | 2018-08-16 | Telefonaktiebolaget Lm Ericsson (Publ) | Methods, client and server relating to a distributed database |
-
2019
- 2019-10-29 CN CN201911037777.4A patent/CN110750497B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160019254A1 (en) * | 2014-07-16 | 2016-01-21 | Facebook. Inc. | Tiered data storage architecture |
| CN106648961A (en) * | 2016-09-27 | 2017-05-10 | 上海爱数信息技术股份有限公司 | Integrated blue-ray disc jukebox backup and archiving method |
| CN106649467A (en) * | 2016-09-27 | 2017-05-10 | 上海爱数信息技术股份有限公司 | Blue-ray disc jukebox archiving management method and system |
| WO2018145739A1 (en) * | 2017-02-08 | 2018-08-16 | Telefonaktiebolaget Lm Ericsson (Publ) | Methods, client and server relating to a distributed database |
Non-Patent Citations (2)
| Title |
|---|
| 严文瑞;曹强;姚杰;谢长生;: "一种面向大容量光盘库的新型文件系统" * |
| 屠要峰;刘辉;张国良;刘春;: "一种分布式缓存系统的关键技术及应用" * |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111736956A (en) * | 2020-06-29 | 2020-10-02 | 苏州浪潮智能科技有限公司 | Container service deployment method, device, equipment and readable storage medium |
| CN111736956B (en) * | 2020-06-29 | 2023-01-10 | 苏州浪潮智能科技有限公司 | Container service deployment method, device, equipment and readable storage medium |
| CN111934723A (en) * | 2020-08-26 | 2020-11-13 | 上海仪电（集团）有限公司中央研究院 | Bypass interception Bluetooth communication device, method and application thereof |
| CN112818059A (en) * | 2021-01-27 | 2021-05-18 | 百果园技术(新加坡)有限公司 | Information real-time synchronization method and device based on container release platform |
| CN113032598A (en) * | 2021-04-12 | 2021-06-25 | 郑州航空工业管理学院 | Image design visual transmission system based on big data |
| CN113032598B (en) * | 2021-04-12 | 2023-07-14 | 郑州航空工业管理学院 | Image design visual conveying system based on big data |
| CN115242677A (en) * | 2021-04-23 | 2022-10-25 | 中国移动通信集团四川有限公司 | Home-wide user state monitoring system, method and device |
| CN115242677B (en) * | 2021-04-23 | 2023-09-01 | 中国移动通信集团四川有限公司 | Home-wide user state monitoring system, method and device |
| CN117076094A (en) * | 2023-10-16 | 2023-11-17 | 中国船舶集团有限公司第七〇七研究所 | Method for concurrently processing multiple tasks of cryptographic operation |
| CN117076094B (en) * | 2023-10-16 | 2024-01-16 | 中国船舶集团有限公司第七〇七研究所 | Method for concurrently processing multiple tasks of cryptographic operation |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110750497B (en) | 2023-09-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110750497A (en) | Data scheduling system | |
| JP6566330B2 (en) | Video editing method | |
| US20160062992A1 (en) | Shared server methods and systems for information storage, access, and security | |
| US20070073766A1 (en) | System, Method, and Computer-Readable Medium for Mobile Media Management | |
| US20180077219A1 (en) | Using Cloud Storage As Temporary Cache For Backup | |
| CN106407078B (en) | Client performance monitoring device and method based on information exchange | |
| WO2022134160A1 (en) | Tamperproof evidence obtaining method, system and apparatus, storage medium, and electronic device | |
| CN110351532B (en) | Video big data cloud platform cloud storage service method | |
| CN104079633A (en) | Method and system for synchronously saving data acquired by terminal to cloud server | |
| CN109587141A (en) | A kind of system and method for remote server evidence obtaining | |
| CN112883011A (en) | Real-time data processing method and device | |
| CN111131786A (en) | Video monitoring storage system applying cloud storage | |
| CN114666554B (en) | Edge gateway cloud service management system | |
| CN114666576A (en) | Public safety edge gateway system | |
| CN101751548A (en) | Identity identification managing method and device thereof | |
| US20090204689A1 (en) | Method and apparatus for remote surveillance of a premises | |
| CN115114236A (en) | Management system and management method for photos and/or videos | |
| EP2210367A1 (en) | Method of managing operations for administration, maintenance and operational upkeep, management entity, and corresponding computer program product | |
| US20040249865A1 (en) | System and method for scheduling and naming for database backup | |
| CN111354112A (en) | Access control system, access method and device of access control equipment and gateway equipment | |
| CN112381471B (en) | Safety production supervision system and method based on intelligent video image sampling | |
| WO2018000524A1 (en) | Backup file method and apparatus | |
| US20040243653A1 (en) | System and method for generating an alarm when an abnormal database backup status occurs | |
| WO2019179050A1 (en) | Call log management method and system, computer device and storage medium | |
| CN111459756A (en) | Log processing method and related equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 266237 5th Floor, Building C, Haike Entrepreneurship Center, Aoshanwei Street Office, Jimo District, Qingdao City, Shandong Province Applicant after: Yihai Luyuan (Shandong) Digital Technology Co.,Ltd. Address before: 266237 5th Floor, Building C, Haike Entrepreneurship Center, Aoshanwei Street Office, Jimo District, Qingdao City, Shandong Province Applicant before: SHANDONG E.HUALU INFORMATION TECHNOLOGY Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |