CN113672579A - File synchronization method based on webservice - Google Patents
File synchronization method based on webservice Download PDFInfo
- Publication number
- CN113672579A CN113672579A CN202110999690.6A CN202110999690A CN113672579A CN 113672579 A CN113672579 A CN 113672579A CN 202110999690 A CN202110999690 A CN 202110999690A CN 113672579 A CN113672579 A CN 113672579A
- Authority
- CN
- China
- Prior art keywords
- data
- client
- file
- server
- service
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 84
- 230000001360 synchronised effect Effects 0.000 claims abstract description 72
- 238000009826 distribution Methods 0.000 claims abstract description 17
- 230000004044 response Effects 0.000 claims abstract description 11
- 238000010276 construction Methods 0.000 claims abstract description 3
- 230000008569 process Effects 0.000 claims description 25
- 238000004891 communication Methods 0.000 claims description 21
- 230000005540 biological transmission Effects 0.000 claims description 15
- 238000012545 processing Methods 0.000 claims description 13
- 238000001514 detection method Methods 0.000 claims description 10
- 238000012544 monitoring process Methods 0.000 claims description 9
- 230000000977 initiatory effect Effects 0.000 claims description 6
- 230000008859 change Effects 0.000 claims description 5
- 230000006870 function Effects 0.000 claims description 4
- 238000012423 maintenance Methods 0.000 claims description 4
- 239000002245 particle Substances 0.000 claims description 4
- 230000002159 abnormal effect Effects 0.000 claims description 3
- 238000004458 analytical method Methods 0.000 claims description 3
- 230000003993 interaction Effects 0.000 claims description 3
- 230000008447 perception Effects 0.000 claims description 3
- 238000003672 processing method Methods 0.000 claims description 3
- 230000008439 repair process Effects 0.000 claims description 3
- 238000005516 engineering process Methods 0.000 abstract description 12
- 230000006798 recombination Effects 0.000 abstract description 2
- 238000005215 recombination Methods 0.000 abstract description 2
- 239000003795 chemical substances by application Substances 0.000 description 8
- 238000012986 modification Methods 0.000 description 7
- 230000004048 modification Effects 0.000 description 7
- 102100029768 Histone-lysine N-methyltransferase SETD1A Human genes 0.000 description 3
- 101000865038 Homo sapiens Histone-lysine N-methyltransferase SETD1A Proteins 0.000 description 3
- 238000011161 development Methods 0.000 description 3
- 239000000463 material Substances 0.000 description 3
- 101150117538 Set2 gene Proteins 0.000 description 2
- 239000000969 carrier Substances 0.000 description 2
- 238000012937 correction Methods 0.000 description 2
- 238000005265 energy consumption Methods 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 239000000203 mixture Substances 0.000 description 2
- 238000003860 storage Methods 0.000 description 2
- 241000422846 Sequoiadendron giganteum Species 0.000 description 1
- 229910000831 Steel Inorganic materials 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 230000003190 augmentative effect Effects 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 239000010959 steel Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 208000024891 symptom Diseases 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/178—Techniques for file synchronisation in file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/30—Authentication, i.e. establishing the identity or authorisation of security principals
- G06F21/44—Program or device authentication
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention relates to a webservice-based file synchronization method, which is constructed based on a distributed service combination engine, and is characterized in that a synchronous backup request is initiated through a client, a file to be synchronized is scheduled and distributed according to a data-perception-based service combination execution scheduling algorithm, an old data file is scheduled and distributed according to an obtained scheduling and distribution result, after all data blocks are compared, difference block information needing to be transmitted is stored in a cache, a response difference data block is sent to a server according to the received cache, and the server performs recombination to complete incremental synchronization. The distributed service combination engine for the big data application is realized, and the service combination technology is applied to the construction of the big data application, so that specific data updating is not required to be synchronized during data synchronization, and only the difference between data before and after the data updating is synchronized, thereby greatly reducing the data volume of the synchronized data, saving the synchronization resources and improving the synchronization efficiency.
Description
Technical Field
The application relates to the technical field of computers, in particular to a file synchronization method based on webservice.
Background
In the economic and high-speed developed society, enterprise communication files are carriers of enterprise identification and external communication information, so that communication files need to be exchanged frequently between people. The current enterprise communication archives generally adopt paper archives, are simple and practical in a certain period, and with the high-speed development of information technology, paper archive carriers are out of date and untimely.
Firstly, the information contained in the paper file is quite limited, mainly is text information with limited quantity, can not support personalized image, language and video declaration information of pictures, and is difficult to completely describe the communication and identity information of individuals;
secondly, the adaptability of the paper file to information change and personalized customization is poor, the paper file needs to be printed again every time of change, and meanwhile, the file holder cannot be informed actively in time when identity information and a communication mode are changed;
thirdly, the paper file can not be quickly searched and searched, and when the paper file is stored for more than 100 sheets, the search becomes very difficult and consumes quite a long time;
fourthly, the paper file does not have the capability of mutual synchronization with other communication equipment, if a manual comparison input method is generally adopted for synchronization, although a scanning input method is adopted, the accuracy rate cannot be guaranteed, later-stage manual correction is needed, and meanwhile, a set of scanning equipment is added, so that the treatment of symptoms is not permanent;
fifth, and most critical, paper archives are not very environmentally friendly, requiring 17 trees to produce 1 ton of paper, as calculated by the relevant experts. In China, paper making consumes over 2 billion trees with more than 20 years in 1 year, energy consumption for producing 1 ton of paper is equivalent to energy consumption for producing 1 ton of steel, for example, the Qingdao in medium cities is 30 tons of paper required by the whole Qingdao in each month for making files, and 600 big trees are needed to be crushed for making the paper. A large amount of paper is wasted for the early-stage archives after the communication information is changed and printed again, 80% of the archives exchanged with each other are in an idle useless state for a long time, a large amount of paper is wasted, and hidden dangers exist in sanitation due to the hand-to-hand exchange mode. Therefore, the paper file is not economical and inconvenient. Meanwhile, a large amount of paper is wasted, and the environment-friendly concept is not met.
Some electronic file products and related exchange technologies also exist in the current market, but none of the electronic file products and the related exchange technologies can be widely popularized and used, and besides market factors, the electronic file products and the related exchange technologies mainly have some defects and shortcomings in the technology:
(1) support for multimedia information is insufficient. In the era of high-speed information development, simple character information cannot completely express the identity and communication information of an individual, and simultaneously, with the expansion of the business, people with the same name and the same name in a communication file are more and more, and the uniqueness of the single character information is difficult to ensure. It is a necessity for the development of technology and demand to adopt text, pictures, sound, video and even other unique information to make enterprise communication files and identity materials.
(2) Support for multiple communication protocols and seamless switching is inadequate. Currently, most electronic file exchange technologies adopt an infrared mode, and the practicability is relatively poor due to poor directivity of an infrared interface; other have also used bluetooth technology but have not solved the remote switching problem. The short-distance exchange is realized through Bluetooth and WIFI, the remote exchange is realized through WIFI and the Internet, a multi-protocol integrated seamless exchange and synchronous platform is formed, and the realization of the conventional product is difficult.
(3) Intelligent switching and active upgrade services cannot be implemented. The traditional products and the technology need a large amount of manual and tedious operations in the process of realizing file exchange, particularly, when files with different data formats are exchanged, a large amount of comparison work and later-stage correction are needed, and the usability is poor; meanwhile, after the information of the enterprise file is modified, the device which stores the file cannot be informed to automatically upgrade.
(4) Security and anti-volatility are insufficient. The traditional products and technologies do not realize strict login authentication and exchange authentication, and can not effectively prevent communication files from being leaked; meanwhile, distributed redundant storage cannot be realized by isolated equipment, and the whole communication file is lost under the condition that the equipment is damaged or lost.
(5) There are few external extensible services. Most of the existing electronic archive products are single handheld devices, the exchange of the electronic archive products mainly occurs between the same type of devices, and the exchange or synchronization between heterogeneous communication devices and systems is almost impossible; meanwhile, the system also does not provide background website service support, can not realize the maintenance, output and exchange of electronic files on WEB, and has little expandable service function support.
The existing enterprise informatization management system cannot integrate, process and analyze all information in the production process of an enterprise, so that an accurate plan and decision can be made, and the timeliness, the accuracy and the effectiveness of the production process control of large-scale customized furniture enterprises are influenced.
Disclosure of Invention
In order to solve the problem of low synchronization efficiency in current archive information management, the invention discloses an archive synchronization method based on web service, which is used for reasonably scheduling the synchronization of archives based on a web service architecture.
The invention relates to a file synchronization method based on webservice, which is applied between a client and a server and is constructed based on a distributed service combination engine, and is characterized by comprising the following steps:
s1, the client side initiates a synchronous backup request;
s2, the client side carries out scheduling distribution on the files to be synchronized according to a service combination execution scheduling algorithm based on data perception;
s3, the client sends the dispatching distribution result to the server through the network transmission module;
s4, the server side carries out scheduling distribution on the old data files according to the same mode according to the obtained scheduling distribution result, compares all the data blocks, stores the difference block information to be transmitted into a cache, and transmits the difference block information to the client side;
and S5, the client sends the response difference data block to the server according to the received cache, and the server recombines the difference information and the old data information to complete the increment synchronization.
Further, the archive synchronization method based on web service according to claim 1, applied between a client and a server, wherein the method is constructed based on a distributed service composition engine, and further comprises:
modeling a user requirement by utilizing a model-driven service combination to obtain a big data service combination model, namely a Directed Acyclic Graph (DAG) flow file, wherein the big data service combination model meets the requirement of the user for constructing big data application through analysis and execution of a distributed service combination engine;
a brand-new 'combination model' which meets the business requirement is reconstructed through the existing service components, the model can be analyzed into a big data service combination process instance through a service combination engine, and the big data service combination process instance is composed of archive data to be synchronized. Each archive data to be synchronized is generated by instantiating a big data service task in a flow file through a service combination engine;
by re-scheduling the execution sequence of the sub-services, the data stream submits a problem request, an intermediate result is obtained, the problem solution of the sub-services is obtained, and the problem solution required by the user is finally fed back;
and each archive data to be synchronized is dispatched to a service agent node to be processed for execution, and by setting the service agent node, a service provider performs data interaction with the nearest service agent node and stores the processed intermediate result data set on the service agent node for use by other archive data to be synchronized.
Further, the archive synchronization method based on webservice is applied between a client and a server, and is characterized in that, the S1, the client initiates a synchronous backup request, and the method further includes:
s101, the client detects the network performance of the current client cluster;
s102, carrying out priority division on the file to be synchronized of the current client based on the detection result of the network performance detection;
s103, initiating a synchronous backup request based on the result of the priority division of the files to be synchronized;
and initiating synchronous backup requests based on the priority division result of the files to be synchronized, wherein the synchronous backup requests are initiated at set time intervals according to the priority sequence.
Further, the method for synchronizing a file based on webservice is applied between a client and a server, and is characterized in that, in S2, the client performs scheduling distribution on the file to be synchronized according to a scheduling algorithm executed by a service combination based on data awareness, and the method further includes:
the method comprises the steps of dividing files to be synchronized into four groups according to service and file data characteristics, wherein the four groups are respectively provided with service constraint and file data intensive type, file data intensive type without service constraint, file data intensive type with service constraint and file data non-intensive type, and file data non-intensive type without service constraint and file data non-intensive type, and realizing dynamic matching of task examples and distributed computing resources in each group based on an improved particle swarm algorithm.
Further, the archive synchronization method based on webservice is applied between a client and a server, and is characterized in that, in S3, the client sends a scheduling assignment result to the server through a network transmission module, and the method further includes:
a TCP/IP link is adopted, and a multi-process mode is used for improving the transmission efficiency;
the initialization starts, firstly, configuration file information is read, and an IP address and network port information are obtained;
the main process is created through createMainProcess () function, which is primarily responsible for network communication links. Executing a main process, monitoring the obtained network port, and waiting for a response message of a server;
when the server side responds to the request from the client side, the main process responds to the request to create a sub-process and is mainly responsible for data transmission at the two ends;
and the sub-process is closed after the communication transmission is finished, and the main process still keeps monitoring the port and waits for the next link.
Further, the archive synchronization method based on webservice is applied between a client and a server, and is characterized in that, in step S4, the server performs scheduling allocation on old data files according to the same manner according to the obtained scheduling allocation result, compares all data blocks, stores difference block information to be transmitted into a cache, and transmits the difference block information to the client, and the method further includes:
in the case that there is a data update in at least one data block created for the scene object, determining updated data in the data block where the data update exists;
under the condition that a preset synchronization condition is met, generating a corresponding difference description data block according to the updated data;
and synchronizing the difference description data block to the client.
Further, the archive synchronization method based on webservice is applied between a client and a server, and is characterized in that, in S5, the client sends a difference data block of a response to a server according to a received cache, and the server reassembles difference information and old data information to complete incremental synchronization, and the method further includes:
monitoring the change condition of the appointed directory or file of the client in real time, and generating a queue by changing the file in an event form when the monitored file changes to wait for processing;
carrying out exception handling on the synchronization process;
the initialization comprises the initialization of required parameters and file paths;
and after receiving the exception message, acquiring the exception type and the detailed exception description of the exception message, and performing different processing according to different exception types.
Recording different exception types and contents into a system log according to the format so as to find and repair exception sources more quickly during checking;
if the abnormal condition is very serious, the existing processing method can not solve the problem, and then the management personnel is informed to carry out detection and maintenance.
The invention relates to a webservice-based file synchronization method, which is constructed based on a distributed service combination engine, and is characterized in that a synchronous backup request is initiated through a client, a file to be synchronized is scheduled and distributed according to a data-perception-based service combination execution scheduling algorithm, an old data file is scheduled and distributed according to an obtained scheduling and distribution result, after all data blocks are compared, difference block information needing to be transmitted is stored in a cache, a response difference data block is sent to a server according to the received cache, and the server performs recombination to complete incremental synchronization. The distributed service combination engine for the big data application is realized, and the service combination technology is applied to the construction of the big data application, so that specific data updating is not required to be synchronized during data synchronization, and only the difference between data before and after the data updating is synchronized, thereby greatly reducing the data volume of the synchronized data, saving the synchronization resources and improving the synchronization efficiency.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
FIG. 1 is a flowchart illustrating a method for synchronizing files based on webservice according to the present invention;
fig. 2 is a flowchart of a first embodiment of a webservice-based file synchronization method according to the present invention.
Detailed Description
Illustrative embodiments of the present application include, but are not limited to, a workflow-based archive scheduling method.
It will be appreciated that as used herein, the terms "module," "unit" may refer to or include an Application Specific Integrated Circuit (ASIC), an electronic circuit, a processor (shared, dedicated, or group) and/or memory that execute one or more software or firmware programs, a combinational logic circuit, and/or other suitable hardware components that provide the described functionality, or may be part of such hardware components.
It is to be appreciated that in various embodiments of the present application, the processor may be a microprocessor, a digital signal processor, a microcontroller, or the like, and/or any combination thereof. According to another aspect, the processor may be a single-core processor, a multi-core processor, the like, and/or any combination thereof.
It is to be appreciated that a workflow-based archive scheduling method provided herein can be implemented on a variety of electronic devices, including, but not limited to, a server, a distributed server cluster of multiple servers, a cell phone, a tablet, a laptop, a desktop computer, a wearable device, a head-mounted display, a mobile email device, a portable game console, a portable music player, a reader device, a personal digital assistant, a virtual reality or augmented reality device, a television or other electronic device having one or more processors embedded or coupled therein, and the like.
Referring to fig. 1, the present invention requests to protect a archive synchronization method based on webservice, which is applied between a client and a server and is constructed based on a distributed service combination engine, and is characterized in that the method includes the following steps:
s1, the client side initiates a synchronous backup request;
s2, the client side carries out scheduling distribution on the files to be synchronized according to a service combination execution scheduling algorithm based on data perception;
s3, the client sends the dispatching distribution result to the server through the network transmission module;
s4, the server side carries out scheduling distribution on the old data files according to the same mode according to the obtained scheduling distribution result, compares all the data blocks, stores the difference block information to be transmitted into a cache, and transmits the difference block information to the client side;
and S5, the client sends the response difference data block to the server according to the received cache, and the server recombines the difference information and the old data information to complete the increment synchronization.
Further, the archive synchronization method based on web service according to claim 1, applied between a client and a server, wherein the method is constructed based on a distributed service composition engine, and further comprises:
modeling a user requirement by utilizing a model-driven service combination to obtain a big data service combination model, namely a Directed Acyclic Graph (DAG) flow file, wherein the big data service combination model meets the requirement of the user for constructing big data application through analysis and execution of a distributed service combination engine;
a brand-new 'combination model' which meets the business requirement is reconstructed through the existing service components, the model can be analyzed into a big data service combination process instance through a service combination engine, and the big data service combination process instance is composed of archive data to be synchronized. Each archive data to be synchronized is generated by instantiating a big data service task in a flow file through a service combination engine;
by re-scheduling the execution sequence of the sub-services, the data stream submits a problem request, an intermediate result is obtained, the problem solution of the sub-services is obtained, and the problem solution required by the user is finally fed back;
further, referring to fig. 2, the archive synchronization method based on webservice according to the present invention is applied between a client and a server, and is characterized in that, in step S1, the client initiates a synchronous backup request, which further includes:
s101, the client detects the network performance of the current client cluster;
s102, carrying out priority division on the file to be synchronized of the current client based on the detection result of the network performance detection;
s103, initiating a synchronous backup request based on the result of the priority division of the files to be synchronized;
and initiating synchronous backup requests based on the priority division result of the files to be synchronized, wherein the synchronous backup requests are initiated at set time intervals according to the priority sequence.
Specifically, a file synchronization request is received, wherein the file synchronization request carries identification information corresponding to data to be synchronized;
acquiring data volume information corresponding to the data to be synchronized from a preset database according to the identification information, wherein the preset database stores identification information and data volume information respectively corresponding to each piece of data, the data volume information is used for identifying the data volume of the data to be synchronized, and the file to be synchronized is identified based on the data volume to perform priority division;
distributing a data processing cluster corresponding to the archive synchronization request according to the data volume information, wherein the data processing cluster comprises one or more processors capable of executing data synchronization operation;
detecting the data synchronization performance corresponding to the processor according to a preset time interval;
and updating the data processing cluster according to the detection result.
By detecting a processor in a data processing cluster, when the processing capability of the processor does not conform to the current cluster, reusing the processor to process a data synchronization task will reduce the task processing efficiency of the entire data processing cluster.
Further, the method for synchronizing a file based on webservice is applied between a client and a server, and is characterized in that, in S2, the client performs scheduling distribution on the file to be synchronized according to a scheduling algorithm executed by a service combination based on data awareness, and the method further includes:
the method comprises the steps of dividing files to be synchronized into four groups according to service and file data characteristics, wherein the four groups are respectively provided with service constraint and file data intensive type, file data intensive type without service constraint, file data intensive type with service constraint and file data non-intensive type, and file data non-intensive type without service constraint and file data non-intensive type, and realizing dynamic matching of task examples and distributed computing resources in each group based on an improved particle swarm algorithm.
And each archive data to be synchronized is dispatched to a service agent node to be processed for execution, and by setting the service agent node, a service provider performs data interaction with the nearest service agent node and stores the processed intermediate result data set on the service agent node for use by other archive data to be synchronized.
The service combination is applied according to the business constraint principle, the archive data to be synchronized is divided into two sets Set1(Set 1: task instance Set with business constraint), Set2(Set 2: task instance Set without business constraint), and the range of related computing nodes is determined for the task instances in the Set1 for subsequent operation.
According to the task example principle, the two sets of the archive data to be synchronized, Set1 and Set2, are further divided according to the data density degree of each Set of the archive data to be synchronized, and the data-dense Set of the archive data to be synchronized is divided into sets Set11 and Set 21. The archive data to be synchronized in sets Set11, Set21 will be distributed to the central compute node resources, dividing the data non-intensive task instances into sets Set12, Set 22. The task instances in Set12, Set22 will be allocated to edge compute node resources.
And according to the capacity constraint principle, the tasks in the sets of the sets 11, 21, 12 and 22 are scheduled to the corresponding computing nodes according to the data dependency and the time sequence dependency. The quantity of the distributed tasks obtained on each computing node is based on the fact that on the premise that the service combination application instance deadline time constraint is met, the overall resource utilization rate of the system is high, and the quantity of the distributed task instances is relatively large when the resource utilization rate is high.
According to the system constraint principle, the load balance of the whole system is monitored, and when too many task instances or a fault node occurs on a computing node, the task instances are dispatched to other computing nodes.
And repeatedly traversing each group of application instances in the service combination application set DSC to obtain each service combination application instance (which can be regarded as directed acyclic graph DAG from the visualization perspective) to obtain a reasonable task instance allocation scheme and deploying the reasonable task instance allocation scheme to the corresponding computing node.
The problem of the mutual mapping of the archival data to be synchronized and the resources is to employ an improved Particle Swarm Optimization (PSO) algorithm to place the archival data to be synchronized onto the appropriate computing resources.
Further, the archive synchronization method based on webservice is applied between a client and a server, and is characterized in that, in S3, the client sends a scheduling assignment result to the server through a network transmission module, and the method further includes:
a TCP/IP link is adopted, and a multi-process mode is used for improving the transmission efficiency;
the initialization starts, firstly, configuration file information is read, and an IP address and network port information are obtained;
the main process is created through createMainProcess () function, which is primarily responsible for network communication links. Executing a main process, monitoring the obtained network port, and waiting for a response message of a server;
when the server side responds to the request from the client side, the main process responds to the request to create a sub-process and is mainly responsible for data transmission at the two ends;
and the sub-process is closed after the communication transmission is finished, and the main process still keeps monitoring the port and waits for the next link.
Further, the archive synchronization method based on webservice is applied between a client and a server, and is characterized in that, in step S4, the server performs scheduling allocation on old data files according to the same manner according to the obtained scheduling allocation result, compares all data blocks, stores difference block information to be transmitted into a cache, and transmits the difference block information to the client, and the method further includes:
in the case that there is a data update in at least one data block created for the scene object, determining updated data in the data block where the data update exists;
under the condition that a preset synchronization condition is met, generating a corresponding difference description data block according to the updated data;
and synchronizing the difference description data block to the client.
Creating at least one corresponding data block aiming at the scene object, receiving a modification instruction aiming at a target data block in the at least one data block, and modifying data stored in the target data block according to the modification instruction.
The server may determine, in a case where there is a data update in at least one data block created for the scene object, updated data in the data block where the data update exists; under the condition that a preset synchronization condition is met, generating a corresponding difference description data block according to the updated data; and synchronizing the difference description data block to the client to be synchronized. In this case, when a certain data in the scene object is updated, a corresponding difference description data block may be generated, where the difference description data block may represent a difference between the updated data and the data before updating, and after the difference description data block is sent to the client to be synchronized, the client to be synchronized may know a difference between the updated data and the data before updating, so that the locally stored data before updating may be modified according to the difference to obtain the updated data. Therefore, when data synchronization is carried out, specific updating data does not need to be synchronized, and only the difference between the data before and after the updating is synchronized, so that the data volume of the data is greatly reduced, the synchronization resource is saved, and the synchronization efficiency is improved.
Further, the archive synchronization method based on webservice is applied between a client and a server, and is characterized in that, in S5, the client sends a difference data block of a response to a server according to a received cache, and the server reassembles difference information and old data information to complete incremental synchronization, and the method further includes:
monitoring the change condition of the appointed directory or file of the client in real time, and generating a queue by changing the file in an event form when the monitored file changes to wait for processing;
carrying out exception handling on the synchronization process;
the initialization comprises the initialization of required parameters and file paths;
and after receiving the exception message, acquiring the exception type and the detailed exception description of the exception message, and performing different processing according to different exception types.
Recording different exception types and contents into a system log according to the format so as to find and repair exception sources more quickly during checking;
if the abnormal condition is very serious, the existing processing method can not solve the problem, and then the management personnel is informed to carry out detection and maintenance.
Those skilled in the art will appreciate that the disclosure of the present disclosure is susceptible to numerous variations and modifications. For example, the various devices or components described above may be implemented in hardware, or may be implemented in software, firmware, or a combination of some or all of the three.
Flow charts are used in this disclosure to illustrate steps of methods according to embodiments of the disclosure. It should be understood that the preceding and following steps are not necessarily performed in the exact order in which they are performed. Rather, various steps may be processed in reverse order or simultaneously. Also, other operations may be added to the processes.
It will be understood by those skilled in the art that all or part of the steps of the above methods may be implemented by instructing the relevant hardware through a computer program, and the program may be stored in a computer readable storage medium, such as a read-only memory, a magnetic or optical disk, and the like. Alternatively, all or part of the steps of the above embodiments may be implemented using one or more integrated circuits. Accordingly, each module/unit in the above embodiments may be implemented in the form of hardware, and may also be implemented in the form of a software functional module. The present disclosure is not limited to any specific form of combination of hardware and software.
Unless otherwise defined, all terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.
The foregoing is illustrative of the present disclosure and is not to be construed as limiting thereof. Although a few exemplary embodiments of this disclosure have been described, those skilled in the art will readily appreciate that many modifications are possible in the exemplary embodiments without materially departing from the novel teachings and advantages of this disclosure. Accordingly, all such modifications are intended to be included within the scope of this disclosure as defined in the claims. It is to be understood that the foregoing is illustrative of the present disclosure and is not to be construed as limited to the specific embodiments disclosed, and that modifications to the disclosed embodiments, as well as other embodiments, are intended to be included within the scope of the appended claims. The present disclosure is defined by the claims and their equivalents.
In the description herein, references to the description of the term "one embodiment," "some embodiments," "an illustrative embodiment," "an example," "a specific example," or "some examples" or the like mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
While embodiments of the invention have been shown and described, it will be understood by those of ordinary skill in the art that: various changes, modifications, substitutions and alterations can be made to the embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.
Claims (7)
1. A archive synchronization method based on webservice is applied between a client and a server and is constructed based on a distributed service combination engine, and is characterized by comprising the following steps:
s1, the client side initiates a synchronous backup request;
s2, the client side carries out scheduling distribution on the files to be synchronized according to a service combination execution scheduling algorithm based on data perception;
s3, the client sends the dispatching distribution result to the server through the network transmission module;
s4, the server side carries out scheduling distribution on the old data files according to the same mode according to the obtained scheduling distribution result, compares all the data blocks, stores the difference block information to be transmitted into a cache, and transmits the difference block information to the client side;
and S5, the client sends the response difference data block to the server according to the received cache, and the server recombines the difference information and the old data information to complete the increment synchronization.
2. The webservice-based archive synchronization method applied between the client and the server as claimed in claim 1, wherein the distributed service combination engine-based construction further comprises:
modeling a user requirement by utilizing a model-driven service combination to obtain a big data service combination model, namely a Directed Acyclic Graph (DAG) flow file, wherein the big data service combination model meets the requirement of the user for constructing big data application through analysis and execution of a distributed service combination engine;
a brand-new 'combination model' which meets the business requirement is reconstructed through the existing service components, the model can be analyzed into a big data service combination process instance through a service combination engine, and the big data service combination process instance is composed of archive data to be synchronized. Each archive data to be synchronized is generated by instantiating a big data service task in a flow file through a service combination engine;
by re-scheduling the execution sequence of the sub-services, the data stream submits a problem request, an intermediate result is obtained, the problem solution of the sub-services is obtained, and the problem solution required by the user is finally fed back;
and each archive data to be synchronized is dispatched to a service agent node to be processed for execution, and by setting the service agent node, a service provider performs data interaction with the nearest service agent node and stores the processed intermediate result data set on the service agent node for use by other archive data to be synchronized.
3. The webservice-based archive synchronization method according to claim 1, applied between a client and a server, wherein the S1, the client initiates a synchronous backup request, and further comprising:
s101, the client detects the network performance of the current client cluster;
s102, carrying out priority division on the file to be synchronized of the current client based on the detection result of the network performance detection;
s103, initiating a synchronous backup request based on the result of the priority division of the files to be synchronized;
and initiating synchronous backup requests based on the priority division result of the files to be synchronized, wherein the synchronous backup requests are initiated at set time intervals according to the priority sequence.
4. The webservice-based archive synchronization method of claim 1, applied between a client and a server, wherein the S2 is for the client to perform scheduling distribution on the archive to be synchronized according to a data-aware-based service combination execution scheduling algorithm, and further comprising:
the method comprises the steps of dividing files to be synchronized into four groups according to service and file data characteristics, wherein the four groups are respectively provided with service constraint and file data intensive type, file data intensive type without service constraint, file data intensive type with service constraint and file data non-intensive type, and file data non-intensive type without service constraint and file data non-intensive type, and realizing dynamic matching of task examples and distributed computing resources in each group based on an improved particle swarm algorithm.
5. The webservice-based archive synchronization method of claim 1, applied between a client and a server, wherein the S3 is for the client to send the scheduling assignment result to the server through a network transmission module, and further comprising:
a TCP/IP link is adopted, and a multi-process mode is used for improving the transmission efficiency;
the initialization starts, firstly, configuration file information is read, and an IP address and network port information are obtained;
the main process is created through createMainProcess () function, which is primarily responsible for network communication links. Executing a main process, monitoring the obtained network port, and waiting for a response message of a server;
when the server side responds to the request from the client side, the main process responds to the request to create a sub-process and is mainly responsible for data transmission at the two ends;
and the sub-process is closed after the communication transmission is finished, and the main process still keeps monitoring the port and waits for the next link.
6. The webservice-based archive synchronization method according to claim 1, applied between a client and a server, wherein the S4, the server performs scheduling allocation on old data files according to the same manner according to the obtained scheduling allocation result, and stores difference block information to be transmitted into a cache after comparing all data blocks, and transmits the difference block information to the client, further comprising:
in the case that there is a data update in at least one data block created for the scene object, determining updated data in the data block where the data update exists;
under the condition that a preset synchronization condition is met, generating a corresponding difference description data block according to the updated data;
and synchronizing the difference description data block to the client.
7. The webservice-based archive synchronization method according to claim 1, applied between a client and a server, wherein the S5, the client sends a difference data block of a response to the server according to the received cache, and the server reassembles the difference information and old data information to complete incremental synchronization, further comprising:
monitoring the change condition of the appointed directory or file of the client in real time, and generating a queue by changing the file in an event form when the monitored file changes to wait for processing;
carrying out exception handling on the synchronization process;
the initialization comprises the initialization of required parameters and file paths;
and after receiving the exception message, acquiring the exception type and the detailed exception description of the exception message, and performing different processing according to different exception types.
Recording different exception types and contents into a system log according to the format so as to find and repair exception sources more quickly during checking;
if the abnormal condition is very serious, the existing processing method can not solve the problem, and then the management personnel is informed to carry out detection and maintenance.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110999690.6A CN113672579B (en) | 2021-08-29 | 2021-08-29 | File synchronization method based on webservice |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110999690.6A CN113672579B (en) | 2021-08-29 | 2021-08-29 | File synchronization method based on webservice |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113672579A true CN113672579A (en) | 2021-11-19 |
| CN113672579B CN113672579B (en) | 2023-11-14 |
Family
ID=78547372
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110999690.6A Active CN113672579B (en) | 2021-08-29 | 2021-08-29 | File synchronization method based on webservice |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113672579B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102624871A (en) * | 2012-02-03 | 2012-08-01 | 杭州杰唐信息技术有限公司 | Remote file synchronizing method realized based on distributed enterprise service bus |
| CN109299056A (en) * | 2018-09-19 | 2019-02-01 | 潍坊工程职业学院 | A kind of method of data synchronization and device based on distributed file system |
-
2021
- 2021-08-29 CN CN202110999690.6A patent/CN113672579B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102624871A (en) * | 2012-02-03 | 2012-08-01 | 杭州杰唐信息技术有限公司 | Remote file synchronizing method realized based on distributed enterprise service bus |
| CN109299056A (en) * | 2018-09-19 | 2019-02-01 | 潍坊工程职业学院 | A kind of method of data synchronization and device based on distributed file system |
Non-Patent Citations (4)
| Title |
|---|
| 何中秋: "一种新的分布式云工作流调度算法", 《数字技术与应用》, vol. 38, no. 10 * |
| 张雨，李芳，周涛: "云计算环境下基于遗传蚁群算法的任务调度研究", 《计算机工程与应用》, vol. 50, no. 6 * |
| 薛 凡: "DAG 分割模型下的云工作流调度策略", 《计算机应用研究》, vol. 36, no. 12 * |
| 黄小庆;关维德;周宇;夏安邦;: "面向服务架构中的服务分类及其应用", 计算机集成制造系统, no. 09 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113672579B (en) | 2023-11-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11494380B2 (en) | Management of distributed computing framework components in a data fabric service system | |
| Varia | Cloud architectures | |
| US20190310977A1 (en) | Bucket data distribution for exporting data to worker nodes | |
| US9276987B1 (en) | Identifying nodes already storing indicated input data to perform distributed execution of an indicated program in a node cluster | |
| US9086923B2 (en) | Autonomic workflow management in dynamically federated, hybrid cloud infrastructures | |
| WO2020258290A1 (en) | Log data collection method, log data collection apparatus, storage medium and log data collection system | |
| US20140149354A1 (en) | High availability for cloud servers | |
| CN110737643B (en) | Big data analysis processing management middle platform based on catering information management system | |
| CN112104723B (en) | Multi-cluster data processing system and method | |
| CN105187512A (en) | Method and system for load balancing of virtual machine clusters | |
| US9218226B2 (en) | System and methods for remote access to IMS databases | |
| US10432703B2 (en) | On-demand session upgrade in a coordination service | |
| CN106095957A (en) | Cross-domain many wave files synchronous method of distributed file system and device | |
| Du et al. | Scientific workflows in IoT environments: a data placement strategy based on heterogeneous edge-cloud computing | |
| Lohrmann et al. | Massively-parallel stream processing under qos constraints with nephele | |
| US20220179711A1 (en) | Method For Platform-Based Scheduling Of Job Flow | |
| US10182104B1 (en) | Automatic propagation of resource attributes in a provider network according to propagation criteria | |
| CN103716397A (en) | Marching method for service-oriented simulation clock | |
| US8812578B2 (en) | Establishing future start times for jobs to be executed in a multi-cluster environment | |
| CN112948088B (en) | Cloud workflow intelligent management and scheduling system in cloud computing platform | |
| US10691653B1 (en) | Intelligent data backfill and migration operations utilizing event processing architecture | |
| WO2022156087A1 (en) | Data blood relationship establishing method and apparatus, computer device, and storage medium | |
| CN112631800A (en) | Kafka-oriented data transmission method and system, computer equipment and storage medium | |
| CN113672579B (en) | File synchronization method based on webservice | |
| CN111241455B (en) | Data processing apparatus, computer device, and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: Room 769, building 2, East Ring Road, Yanqing Park, Zhongguancun, Yanqing District, Beijing 102101 Applicant after: ZHONGDUN innovative digital technology (Beijing) Co.,Ltd. Address before: Room 769, building 2, East Ring Road, Yanqing Park, Zhongguancun, Yanqing District, Beijing 102101 Applicant before: ZHONGDUN innovation archives management (Beijing) Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |