CN113704212A - Data synchronization method, device and equipment of server and computer storage medium - Google Patents
Data synchronization method, device and equipment of server and computer storage medium Download PDFInfo
- Publication number
- CN113704212A CN113704212A CN202010445601.9A CN202010445601A CN113704212A CN 113704212 A CN113704212 A CN 113704212A CN 202010445601 A CN202010445601 A CN 202010445601A CN 113704212 A CN113704212 A CN 113704212A
- Authority
- CN
- China
- Prior art keywords
- processing
- server
- file information
- file
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 75
- 238000012545 processing Methods 0.000 claims abstract description 423
- 230000006870 function Effects 0.000 claims description 21
- 238000003780 insertion Methods 0.000 claims description 9
- 230000037431 insertion Effects 0.000 claims description 9
- 238000012217 deletion Methods 0.000 claims description 7
- 230000037430 deletion Effects 0.000 claims description 7
- 238000003672 processing method Methods 0.000 claims description 4
- 230000008569 process Effects 0.000 abstract description 30
- 230000001360 synchronised effect Effects 0.000 abstract description 30
- 238000004891 communication Methods 0.000 description 6
- 238000007405 data analysis Methods 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 4
- 238000012423 maintenance Methods 0.000 description 4
- 230000002159 abnormal effect Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 230000001960 triggered effect Effects 0.000 description 2
- 238000011144 upstream manufacturing Methods 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/178—Techniques for file synchronisation in file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Computing Systems (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data synchronization method of a server, which comprises the following steps: the client receives the synchronous instruction and acquires a processing mode corresponding to the synchronous instruction; acquiring a data channel corresponding to the processing mode in the server; and sending the file information corresponding to the synchronization instruction to the server through the acquired data channel. The invention also discloses a data synchronization method, a device, equipment and a computer storage medium of the server, wherein the data channel corresponding to the processing mode is determined by acquiring the processing mode of data synchronization, and the file information to be synchronized is sent to the server through the corresponding data channel, so that the server directly carries out data synchronization according to the processing mode corresponding to the data channel, the processing mode is determined without analyzing the file information to be synchronized, the data processing amount of the server in the data synchronization process is reduced, and the data synchronization efficiency is improved.
Description
Technical Field
The present invention relates to the field of data synchronization technologies, and in particular, to a data synchronization method, apparatus, device, and computer storage medium for a server.
Background
An Elastic Search (ES) is a distributed, highly-extended, highly-real-time search and data analysis engine, is used for providing a full-text search function with distributed multi-user capability, and is a popular enterprise-level search engine. The ES is generally used in cloud computing, and has the advantages of real-time search, stability, reliability, convenience in installation and use and the like. If a client accesses network resources in the ES server, a large number of log files are usually generated in the client, and the client needs to periodically send the log files to the server for the ES server to perform data analysis according to the log files.
At present, in the process of sending a log file to an ES server, the log file and a specified file processing mode are usually packed and sent to the ES server for the ES server to analyze data, and process the log file according to the specified file processing mode, but the ES server generally receives a large number of log files sent by a plurality of clients at the same time, so that the data processing amount in the analyzing process is very large, the performance bottleneck of the ES server is reached, and the efficiency of data synchronization is low.
The above is only for the purpose of assisting understanding of the technical aspects of the present invention, and does not represent an admission that the above is prior art.
Disclosure of Invention
The invention mainly aims to provide a data synchronization method, a data synchronization device, data synchronization equipment and a computer storage medium for a server, which aim to send file information through a data channel corresponding to a processing mode, so that the server does not need to determine the processing mode by analyzing the file information, and the data synchronization efficiency is improved.
In order to achieve the above object, the present invention provides a data synchronization method for a server, including the following steps:
the method comprises the steps that a client receives a synchronization instruction and obtains a processing mode corresponding to the synchronization instruction, wherein the processing mode comprises any one of updating processing, inserting processing, deleting processing and creating processing;
acquiring a data channel corresponding to the processing mode in the server;
and sending the file information corresponding to the synchronization instruction to the server through the acquired data channel, wherein the server is used for processing the received file corresponding to the file information according to the processing mode corresponding to the data channel.
Optionally, the step of obtaining the processing mode corresponding to the synchronization instruction includes:
determining a synchronization type of the synchronization instruction;
and acquiring the processing mode corresponding to the synchronization type.
In order to achieve the above object, the present invention provides a data synchronization method for a server, including the following steps:
receiving file information sent by a client;
acquiring a data channel corresponding to the file information;
acquiring a processing mode corresponding to the data channel, wherein the processing mode comprises any one of updating processing, inserting processing, deleting processing and creating processing;
and processing the file corresponding to the file information according to the processing mode.
Optionally, the step of processing the file corresponding to the file information according to the processing manner includes:
detecting the state of a processing thread corresponding to the processing mode;
if the processing thread is in a non-running state, acquiring a logic value of a preset configuration parameter of the processing thread;
if the logic value is a preset logic value, starting the processing thread;
and processing the file corresponding to the file information through the processing thread.
Optionally, after the step of starting the processing thread if the logical value is a preset logical value, the data synchronization method of the server further includes:
after the processing thread is started, detecting whether an index function of the processing thread is available;
and if the index function is available, executing the step of processing the file corresponding to the file information through the processing thread.
Optionally, after the step of processing the file corresponding to the file information according to the processing manner, the method further includes:
after processing the file corresponding to the file information, acquiring a processing result of processing the file corresponding to the file information;
if the processing result is processing failure, acquiring the processing times of processing the file corresponding to the file information;
if the processing times are less than the preset times, returning to the step of executing the processing of the file corresponding to the file information according to the processing mode;
and if the processing times are greater than or equal to the preset times, generating and storing a failure record corresponding to the processing result.
Optionally, after the step of obtaining a processing result of processing the file corresponding to the file information, the step of the data synchronization method of the server further includes:
if the processing result is processing failure, acquiring a failure type corresponding to the processing result;
if the failure type is a data synchronization error, executing the step of acquiring the processing times for processing the file corresponding to the file information;
and if the failure type is a data format error, generating and storing a failure record corresponding to the processing result.
In addition, to achieve the above object, the present invention provides a data synchronization apparatus for a server, including:
the system comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for receiving a synchronization instruction and acquiring a processing mode corresponding to the synchronization instruction, and the processing mode comprises any one of updating processing, inserting processing, deleting processing and creating processing;
the acquisition module is further configured to acquire a data channel corresponding to the processing mode in the server;
and the sending module is used for sending the file information corresponding to the synchronization instruction to the server through the acquired data channel, wherein the server is used for processing the received file corresponding to the file information according to the processing mode corresponding to the data channel.
In addition, to achieve the above object, the present invention provides a data synchronization apparatus for a server, including:
the receiving module is used for receiving file information sent by the client;
the acquisition module is used for acquiring a data channel corresponding to the file information;
the obtaining module is further configured to obtain a processing mode corresponding to the data channel, where the processing mode includes any one of update processing, insertion processing, deletion processing, and creation processing;
and the processing module is used for processing the file corresponding to the file information according to the processing mode.
In addition, to achieve the above object, the present invention further provides a data synchronization apparatus of a server, including: a memory, a processor and a data synchronization program of a server stored on the memory and operable on the processor, the data synchronization program of the server when executed by the processor implementing the steps of the data synchronization method of the server as described in any one of the above.
Further, to achieve the above object, the present invention also provides a computer storage medium having stored thereon a data synchronization program of a server, which when executed by a processor, implements the steps of the data synchronization method of the server as described in any one of the above.
According to the data synchronization method, the data synchronization device, the data synchronization equipment and the computer storage medium of the server provided by the embodiment of the invention, a client receives a synchronization instruction, acquires a processing mode corresponding to the synchronization instruction, wherein the processing mode comprises any one of updating processing, inserting processing, deleting processing and creating processing, acquires a data channel corresponding to the processing mode in the server, and sends file information corresponding to the synchronization instruction to the server through the acquired data channel, and the server is used for processing a file corresponding to the received file information according to the processing mode corresponding to the data channel. According to the data synchronization method and device, the data channel corresponding to the processing mode is determined by obtaining the processing mode of data synchronization, and the file information to be synchronized is sent to the server through the corresponding data channel, so that the server can directly perform data synchronization according to the processing mode corresponding to the data channel, the processing mode is determined without analyzing the file information to be synchronized, the data processing amount of the server in the data synchronization process is reduced, and the data synchronization efficiency is improved.
Drawings
Fig. 1 is a schematic terminal structure diagram of a hardware operating environment according to an embodiment of the present invention;
FIG. 2 is a flowchart illustrating a data synchronization method of a server according to a first embodiment of the present invention;
FIG. 3 is a flowchart illustrating a data synchronization method of a server according to a second embodiment of the present invention;
FIG. 4 is a flowchart illustrating a data synchronization method of a server according to a third embodiment of the present invention;
FIG. 5 is a flowchart illustrating a fourth embodiment of a data synchronization method of a server according to the present invention;
fig. 6 is an interaction diagram of the data synchronization method of the server according to the present invention.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The embodiment of the invention provides a solution, which is characterized in that a data channel corresponding to a processing mode is determined by acquiring the processing mode of data synchronization, and file information to be synchronized is sent to a server through the corresponding data channel, so that the server directly performs data synchronization according to the processing mode corresponding to the data channel, the processing mode is not required to be determined by analyzing the file information to be synchronized, the data processing amount of the server in the data synchronization process is reduced, and the data synchronization efficiency is improved.
As shown in fig. 1, fig. 1 is a schematic terminal structure diagram of a hardware operating environment according to an embodiment of the present invention.
The terminal in the embodiment of the invention is a Personal Computer (PC), a server, a cloud server and other terminal equipment.
As shown in fig. 1, the terminal may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, a communication bus 1002. Wherein a communication bus 1002 is used to enable connective communication between these components. The user interface 1003 may include a Display screen (Display), an input unit such as a Keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface, a wireless interface. The network interface 1004 may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface). The memory 1005 may be a high-speed RAM memory or a non-volatile memory (e.g., a magnetic disk memory). The memory 1005 may alternatively be a storage device separate from the processor 1001.
Those skilled in the art will appreciate that the terminal structure shown in fig. 1 is not intended to be limiting and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components.
As shown in fig. 1, a memory 1005, which is a kind of computer storage medium, may include therein a data synchronization program of an operating system, a network communication module, a user interface module, and a server.
In the terminal shown in fig. 1, the network interface 1004 is mainly used for connecting to a backend server and performing data communication with the backend server; the user interface 1003 is mainly used for connecting a client (user side) and performing data communication with the client; and the processor 1001 may be configured to invoke a data synchronization program of the server stored in the memory 1005 and perform the following operations:
the method comprises the steps that a client receives a synchronization instruction and obtains a processing mode corresponding to the synchronization instruction, wherein the processing mode comprises any one of updating processing, inserting processing, deleting processing and creating processing;
acquiring a data channel corresponding to the processing mode in the server;
and sending the file information corresponding to the synchronization instruction to the server through the acquired data channel, wherein the server is used for processing the received file corresponding to the file information according to the processing mode corresponding to the data channel.
Further, the processor 1001 may call the data synchronization program of the server stored in the memory 1005, and also perform the following operations:
determining a synchronization type of the synchronization instruction;
and acquiring the processing mode corresponding to the synchronization type.
The processor 1001 may also be configured to invoke a data synchronization program of a server stored in the memory 1005 and perform the following operations:
receiving file information sent by a client;
acquiring a data channel corresponding to the file information;
acquiring a processing mode corresponding to the data channel, wherein the processing mode comprises any one of updating processing, inserting processing, deleting processing and creating processing;
and processing the file corresponding to the file information according to the processing mode.
Further, the processor 1001 may call the data synchronization program of the server stored in the memory 1005, and also perform the following operations:
detecting the state of a processing thread corresponding to the processing mode;
if the processing thread is in a non-running state, acquiring a logic value of a preset configuration parameter of the processing thread;
if the logic value is a preset logic value, starting the processing thread;
and processing the file corresponding to the file information through the processing thread.
Further, the processor 1001 may call the data synchronization program of the server stored in the memory 1005, and also perform the following operations:
after the processing thread is started, detecting whether an index function of the processing thread is available;
and if the index function is available, executing the step of processing the file corresponding to the file information through the processing thread.
Further, the processor 1001 may call the data synchronization program of the server stored in the memory 1005, and also perform the following operations:
after processing the file corresponding to the file information, acquiring a processing result of processing the file corresponding to the file information;
if the processing result is processing failure, acquiring the processing times of processing the file corresponding to the file information;
if the processing times are less than the preset times, returning to the step of executing the processing of the file corresponding to the file information according to the processing mode;
and if the processing times are greater than or equal to the preset times, generating and storing a failure record corresponding to the processing result.
Further, the processor 1001 may call the data synchronization program of the server stored in the memory 1005, and also perform the following operations:
if the processing result is processing failure, acquiring a failure type corresponding to the processing result;
if the failure type is a data synchronization error, executing the step of acquiring the processing times for processing the file corresponding to the file information;
and if the failure type is a data format error, generating and storing a failure record corresponding to the processing result.
Referring to fig. 2, in an embodiment, the data synchronization method of the server includes the following steps:
step S10, the client receives the synchronization instruction, and obtains the processing mode corresponding to the synchronization instruction, wherein the processing mode includes any one of update processing, insert processing, delete processing and create processing;
in this embodiment, the execution body is a client. And the client receives a synchronization instruction triggered by a user and acquires a processing mode corresponding to the synchronization instruction. The user synchronizes the data to the server, and generally performs processes such as updating, inserting, deleting, creating, and the like on the data, so the processing mode may also include any one of the updating process, the inserting process, the deleting process, and the creating process, and the specific processing mode may be determined according to a synchronization instruction triggered by the user.
Optionally, in the step of acquiring the processing manner corresponding to the synchronization instruction, the synchronization type of the synchronization instruction may be determined, and the processing manner corresponding to the synchronization type is acquired. The synchronization instruction may include file data to be synchronized and a synchronization type, and which processing is performed on the file data may be determined according to the synchronization type. The synchronization types generally include update synchronization, insert synchronization, delete synchronization, and create synchronization, and correspond to processing manners.
Step S20, acquiring a data channel corresponding to the processing mode in the server;
in this embodiment, the server is provided with a plurality of data channels, and different data channels correspond to different storage areas or storage paths, and the server can perform different processing on data passing through different data channels, that is, the data channels correspond to processing modes. The corresponding relation is stored in the client in advance, so that the client can obtain the data channel corresponding to the processing mode according to the pre-stored corresponding relation after obtaining the processing mode.
Step S30, sending the file information corresponding to the synchronization instruction to the server through the obtained data channel, where the server is configured to process the file corresponding to the received file information according to the processing manner corresponding to the data channel.
In this embodiment, the client obtains file information corresponding to the synchronization instruction, where the file information is the file to be synchronized itself or an identity corresponding to the file to be synchronized. The client sends the file information to the server through the determined data channel so that the server receives the file information and processes the file corresponding to the file information according to the processing mode corresponding to the data channel for receiving the file information, and synchronization of data from the client to the server is achieved. The client does not need to send the appointed processing mode to the server, the server does not need to analyze the file information and determine the processing mode, the appointed processing mode can be known according to the data channel corresponding to the file information, the data transmission quantity of the client and the data analysis quantity of the server are reduced, and the data synchronization efficiency is improved.
Optionally, a message middleware is provided in the server, and the client sends the file information to the server, that is, sends the file information to the message middleware. For example, if the message middleware is a Kafka cluster, the Kafka cluster includes multiple brokers, multiple topics are stored in the brokers, a storage area of each topic corresponds to one data channel, and the client sends file information through the data channel, that is, the file information is sent to the topic corresponding to the data channel for storage, so that the server performs data synchronization according to a processing mode corresponding to the topic.
Optionally, the client acquires file information corresponding to the synchronization instruction, determines whether to use the file to be synchronized or the identity corresponding to the file to be synchronized as file information according to a processing mode corresponding to the synchronization instruction, and sends the file information to the server. Specifically, if the processing mode is any one of the update processing, the insertion processing, and the creation processing, the file to be synchronized itself is used as the file information, so as to implement data synchronization of the file to be synchronized. If the processing mode is deletion processing, the identity corresponding to the file to be synchronized is used as file information, wherein if the identity corresponding to the file to be synchronized is obtained, the file to be synchronized is also stored in the ES database in advance because the user needs to delete the file to be synchronized, and the client can send the file to be synchronized to the ES database for query so that the ES database feeds back the identity corresponding to the file to be synchronized. Generally, the identity is storage location information of the file to be synchronized in the ES database, or the number of the file to be synchronized in the ES database.
In the technical scheme disclosed in this embodiment, the data channel corresponding to the processing mode is determined by obtaining the processing mode of data synchronization, and the file information to be synchronized is sent to the server through the corresponding data channel, so that the server directly performs data synchronization according to the processing mode corresponding to the data channel, the processing mode does not need to be determined by analyzing the file information to be synchronized, the data processing amount of the server in the data synchronization process is reduced, and the efficiency of data synchronization is improved.
In another embodiment, as shown in fig. 3, based on the embodiment shown in fig. 2, the data synchronization method of the server further includes:
step S40, receiving the file information sent by the client;
step S50, acquiring a data channel corresponding to the file information;
in this embodiment, the execution subject is a server. And if the server receives the file information sent by the client, acquiring a data channel receiving the file information. The server is provided with a plurality of data channels in advance, and different data channels correspond to different storage areas or storage paths, so that if the file information sent by the client is received, the data channel corresponding to the file information can be determined according to the storage area or the storage path of the file information.
Optionally, a message middleware is provided in the server, and the file information sent by the client is received through the message middleware. For example, as shown in fig. 6, if the message middleware is a Kafka cluster, the client pushes the file information to the Kafka cluster, so that the processing thread of the server pulls the file information from the Kafka cluster, and synchronizes the file corresponding to the file information to the ES database in the server according to the processing mode. Kafka is a high-throughput distributed publish-subscribe messaging system that can handle all the action flow data of a consumer in a web site. Specifically, the kafka cluster includes multiple brokers, multiple topics are stored in the brokers, a storage area of each topic corresponds to one data channel, and the client sends file information through the data channel, that is, sends the file information to the topic corresponding to the data channel for storage, so that the processing thread can pull the file information and perform data synchronization according to the processing mode corresponding to the data channel.
Optionally, the multiple brookers in the Kafk cluster are all managed by the Zookeeper, and the multiple brookers determine the main brooker and the standby brooker through an election mechanism, and the Zookeeper can receive file information sent by the client and store the file information to the main brooker and can be used by the processing thread to pull the file information, and the Zookeeper can monitor and acquire data stored on the standby brooker. In the process of electing the main broker from the multiple brokers, the Zookeeper opens only one temporary node at the same time, the multiple brokers can register the temporary node, but only one broker can successfully register the temporary node, other brokers can fail to register, the broker which successfully registers is the main broker, and the broker which fails to register is the standby broker.
Step S60, acquiring a processing mode corresponding to the data channel, wherein the processing mode includes any one of update processing, insertion processing, deletion processing and creation processing;
and step S70, processing the file corresponding to the file information according to the processing mode.
In this embodiment, the server is preset with a corresponding relationship between the data channel and the processing method, and the corresponding relationship is also stored in the client in advance. Therefore, after the server acquires the data channel corresponding to the file information, the processing mode corresponding to the data channel can be acquired according to the pre-stored corresponding relation, and the processing mode corresponding to the data channel is the data synchronization mode designated by the user. The processing mode includes any one of update processing, insertion processing, deletion processing, and creation processing.
Optionally, in the process of processing the file corresponding to the file information according to the processing manner, if the processing manner is an update processing, the file corresponding to the file information in the ES database is acquired, and the file is replaced with the file information. And if the processing mode is an insertion processing, acquiring a file corresponding to the file information in the ES database, and inserting the file information into the file, if the processing mode is a deletion processing, acquiring a file corresponding to the file information in the ES database, and deleting the file, and if the processing mode is a creation processing, storing the file information into the ES database. If the processing mode is insertion processing, if multiple file information exists at the same time, batch insertion of data can be achieved through a bulk insert function, and data synchronization efficiency is improved.
Optionally, after the file corresponding to the file information is processed according to the processing mode, data cleaning, data integration and other processing may be periodically performed on the data stored in the ES database, so as to ensure performance and stability of the ES database.
In the technical scheme disclosed in this embodiment, the server acquires the data channel corresponding to the received file information, determines the processing mode corresponding to the data channel, and processes the file corresponding to the file information according to the processing mode, so that data synchronization can be achieved, data does not need to be analyzed to acquire the processing mode, the data analysis amount of the server is reduced, and the data synchronization efficiency is improved.
In yet another embodiment, as shown in fig. 4, on the basis of the embodiment shown in any one of fig. 2 to 3, the step S70 includes:
step S71, detecting the state of the processing thread corresponding to the processing mode;
in this embodiment, after the server processes the file corresponding to the file information according to the processing manner, the state of the thread used for data synchronization in the server may be continuously monitored, so as to ensure that the data can be normally synchronized into a flexible search (ES) database. The thread for data synchronization comprises a processing thread corresponding to a processing mode.
Optionally, there are multiple threads for data synchronization in the server, and each thread corresponds to one processing method, and each processing method may correspond to multiple threads, for example, a delete thread corresponds to a delete process, and a delete process corresponds to multiple delete threads. The server is provided with a thread maintenance table, the thread maintenance table comprises threads for data synchronization, and the server traverses the thread maintenance table to detect whether each thread in the thread maintenance table is in a running state.
Step S72, if the processing thread is in a non-running state, acquiring a logic value of a preset configuration parameter of the processing thread;
step S73, if the logic value is a preset logic value, starting the processing thread;
step S74, the processing thread processes the file corresponding to the file information.
In this embodiment, if the server detects that the processing thread is in the non-running state, it indicates that the processing thread has exited, so that it can further detect whether the processing thread exits abnormally. Specifically, a logic value of a preset configuration parameter of a processing thread is obtained, whether the processing thread is an abnormal exit is judged according to whether the logic value is the preset logic value, and if the processing thread is the abnormal exit, the processing thread is started so as to process a file corresponding to file information through the processing thread.
Optionally, the preset configuration parameter item of the processing thread is active close (active close), the preset logic value is a flash, and if the logic value of the active close is also a flash, it indicates that the processing thread is an abnormal exit, so the processing thread is started.
Optionally, if the logical value of the active close is true, it indicates that the processing thread is actively closed, so as to detect whether an index function of the processing thread is available, and if the index function is available, it indicates that the processing thread can be normally connected to the ES database, so that the processing thread can be started, and the logical value of the active close is modified into a flash, so as to prevent the processing thread from being actively closed, so as to process the file corresponding to the file information through the processing thread. If the index function is not available, the processing thread cannot be normally connected to the ES database, so the detection of the processing thread can be skipped until the index function is available, and the processing thread is started.
Optionally, the processing thread exits for an exception, and after the processing thread is restarted, it may further be detected whether an index function of the processing thread is available, and if the index function is available, the file corresponding to the file information is processed through the processing thread.
Optionally, if the server detects that the processing thread is in the running state, detecting whether an index function of the processing thread is available, if the index function is available, processing the file corresponding to the file information through the processing thread, if the index function is unavailable, actively closing the processing thread, and setting a logic value of active close to true, which indicates that the processing thread is actively closed, until the index function is detected to be available, and then starting the processing thread.
Optionally, if the message middleware of the server is a Kafka cluster, the connection state between the processing thread and the Zookeeper in the Kafka cluster may also be monitored, so as to ensure the normal connection between the processing thread and the Zookeeper. If the thread and the Zookeeper are in a connection failure state, the processing thread can periodically request connection with the Zookeeper, and if the connection is successfully established, the connection duration is recorded. If the thread and the Zookeeper are in a successful connection state, acquiring the current running time of the processing thread and the connection duration of the processing thread and the Zookeeper, and if the current running time and the connection duration are greater than a preset time interval, indicating that the data transmission of the Zookeeper and the processing thread is asynchronous, and therefore actively closing the processing thread after the processing thread finishes processing the file corresponding to the current file information to avoid data transmission errors.
In the technical scheme disclosed in this embodiment, normal operation of a processing thread is ensured by monitoring the processing thread, if the index function of the processing thread is unavailable, the processing thread is actively closed to avoid that the processing thread cannot be normally connected to a downstream ES database, the processing thread directly stores file information into a local disk of a server, the problems of synchronous data loss and unreliable data of the ES database are avoided, the time length of the processing thread and the time length of an upstream Zookeeper are monitored and compared, data transmission errors are avoided, and stopping and recovering data synchronization at a task flow level are realized by automatically sensing the connection state of upstream and downstream components, so that the reliability of the whole data synchronization process is improved.
In another embodiment, as shown in fig. 5, on the basis of the embodiment shown in any one of fig. 2 to 4, after step S70, the method further includes:
step S80, after processing the file corresponding to the file information, acquiring a processing result of processing the file corresponding to the file information;
in this embodiment, after processing the file corresponding to the file information according to the processing mode, the server synchronously obtains a processing result of processing the file corresponding to the file information, and determines whether the file is successfully synchronized to the ES server according to the processing result.
Step S90, if the processing result is processing failure, acquiring the processing times of processing the file corresponding to the file information;
step S100, if the processing times are less than the preset times, returning to the step of processing the file corresponding to the file information according to the processing mode;
and step S110, if the processing times are more than or equal to the preset times, generating and storing a failure record corresponding to the processing result.
In this embodiment, if the processing result is processing failure, it indicates that the synchronization of the file is failed, so a preset number of times is set, so that the server retries the synchronization of the file for multiple times according to the preset number of times, so as to make the data synchronization of the server more reliable. Specifically, if the processing result is processing failure, the processed times of processing the file corresponding to the file information at the current time are obtained, i.e. retry times, if the processing times is less than the preset times, returning to the step of processing the file corresponding to the file information according to the processing mode, that is, the file corresponding to the file information is reprocessed according to a certain retry period to retry again, if the processing number is greater than or equal to the preset number, it indicates that the retry of the preset number has been performed, but the data synchronization still fails, and therefore, generating a failure record according to the processing result, storing the failure record into a specific index in the ES database, and the user can perform analysis and processing subsequently according to the failure record, so that the server data synchronization function can be maintained conveniently, wherein the failure record comprises error reasons, state codes, file information and the like.
Optionally, if the processing result is a processing failure, a failure type corresponding to the processing result may also be obtained, and if the failure type is a data synchronization error, it indicates that the file itself has no problem, but is a data synchronization error caused by an accidental factor. If the failure type is a data format error, the error of the format of the file is indicated, and the error cannot be solved even if the file is retried for multiple times, so that the failure record corresponding to the processing result can be generated and stored, and the retry of data synchronization is not performed. Whether to retry the data synchronization is determined by identifying the type of the processing result, the steps of the data synchronization are simplified, and the efficiency of the data synchronization is improved.
In the technical solution disclosed in this embodiment, after processing the file corresponding to the file information, a corresponding processing result is obtained, and if the processing result is a processing failure, the step of processing the file corresponding to the file information according to the processing mode is returned to execute to process the file again, so that the data synchronization function of the server is more reliable.
In addition, an embodiment of the present invention further provides a data synchronization apparatus for a server, where the data synchronization apparatus for the server includes:
the system comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for receiving a synchronization instruction and acquiring a processing mode corresponding to the synchronization instruction, and the processing mode comprises any one of updating processing, inserting processing, deleting processing and creating processing;
the acquisition module is further configured to acquire a data channel corresponding to the processing mode in the server;
and the sending module is used for sending the file information corresponding to the synchronization instruction to the server through the acquired data channel, wherein the server is used for processing the received file corresponding to the file information according to the processing mode corresponding to the data channel.
In the technical scheme disclosed in this embodiment, the data channel corresponding to the processing mode is determined by obtaining the processing mode of data synchronization, and the file information to be synchronized is sent to the server through the corresponding data channel, so that the server directly performs data synchronization according to the processing mode corresponding to the data channel, the processing mode does not need to be determined by analyzing the file information to be synchronized, the data processing amount of the server in the data synchronization process is reduced, and the efficiency of data synchronization is improved.
In addition, an embodiment of the present invention further provides a data synchronization apparatus for a server, where the data synchronization apparatus for the server includes:
the receiving module is used for receiving file information sent by the client;
the acquisition module is used for acquiring a data channel corresponding to the file information;
the obtaining module is further configured to obtain a processing mode corresponding to the data channel, where the processing mode includes any one of update processing, insertion processing, deletion processing, and creation processing;
and the processing module is used for processing the file corresponding to the file information according to the processing mode.
In the technical scheme disclosed in this embodiment, the data channel corresponding to the received file information is acquired, the processing mode corresponding to the data channel is determined, and the file corresponding to the file information is processed according to the processing mode, so that data synchronization can be realized, the data does not need to be analyzed to acquire the processing mode, the data analysis amount of the server is reduced, and the data synchronization efficiency is improved.
In addition, an embodiment of the present invention further provides a data synchronization device of a server, where the data synchronization device of the server includes: the data synchronization program of the server is stored on the memory and can run on the processor, and when being executed by the processor, the data synchronization program of the server realizes the steps of the data synchronization method of the server according to the above embodiments.
In addition, an embodiment of the present invention further provides a computer storage medium, where a data synchronization program of a server is stored on the computer storage medium, and the data synchronization program of the server, when executed by a processor, implements the steps of the data synchronization method of the server according to the above embodiments.
Optionally, the data synchronization program adopts a springboot framework.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or system that comprises the element.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) as described above and includes instructions for enabling a terminal device (e.g., a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method according to the embodiments of the present invention.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (11)
1. A data synchronization method of a server is characterized by comprising the following steps:
the method comprises the steps that a client receives a synchronization instruction and obtains a processing mode corresponding to the synchronization instruction, wherein the processing mode comprises any one of updating processing, inserting processing, deleting processing and creating processing;
acquiring a data channel corresponding to the processing mode in the server;
and sending the file information corresponding to the synchronization instruction to the server through the acquired data channel, wherein the server is used for processing the received file corresponding to the file information according to the processing mode corresponding to the data channel.
2. The data synchronization method of the server according to claim 1, wherein the step of obtaining the processing mode corresponding to the synchronization instruction comprises:
determining a synchronization type of the synchronization instruction;
and acquiring the processing mode corresponding to the synchronization type.
3. A data synchronization method of a server is characterized by comprising the following steps:
receiving file information sent by a client;
acquiring a data channel corresponding to the file information;
acquiring a processing mode corresponding to the data channel, wherein the processing mode comprises any one of updating processing, inserting processing, deleting processing and creating processing;
and processing the file corresponding to the file information according to the processing mode.
4. The data synchronization method of the server according to claim 3, wherein the step of processing the file corresponding to the file information according to the processing mode includes:
detecting the state of a processing thread corresponding to the processing mode;
if the processing thread is in a non-running state, acquiring a logic value of a preset configuration parameter of the processing thread;
if the logic value is a preset logic value, starting the processing thread;
and processing the file corresponding to the file information through the processing thread.
5. The method for data synchronization of a server according to claim 4, wherein after the step of starting the processing thread if the logic value is a predetermined logic value, the method for data synchronization of a server further comprises:
after the processing thread is started, detecting whether an index function of the processing thread is available;
and if the index function is available, executing the step of processing the file corresponding to the file information through the processing thread.
6. The data synchronization method of the server according to claim 3, wherein after the step of processing the file corresponding to the file information according to the processing method, the method further comprises:
after processing the file corresponding to the file information, acquiring a processing result of processing the file corresponding to the file information;
if the processing result is processing failure, acquiring the processing times of processing the file corresponding to the file information;
if the processing times are less than the preset times, returning to the step of executing the processing of the file corresponding to the file information according to the processing mode;
and if the processing times are greater than or equal to the preset times, generating and storing a failure record corresponding to the processing result.
7. The data synchronization method of the server according to claim 6, wherein after the step of obtaining the processing result of processing the file corresponding to the file information, the data synchronization method of the server further comprises:
if the processing result is processing failure, acquiring a failure type corresponding to the processing result;
if the failure type is a data synchronization error, executing the step of acquiring the processing times for processing the file corresponding to the file information;
and if the failure type is a data format error, generating and storing a failure record corresponding to the processing result.
8. A data synchronization apparatus of a server, the data synchronization apparatus of the server comprising:
the system comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for receiving a synchronization instruction and acquiring a processing mode corresponding to the synchronization instruction, and the processing mode comprises any one of updating processing, inserting processing, deleting processing and creating processing;
the acquisition module is further configured to acquire a data channel corresponding to the processing mode in the server;
and the sending module is used for sending the file information corresponding to the synchronization instruction to the server through the acquired data channel, wherein the server is used for processing the received file corresponding to the file information according to the processing mode corresponding to the data channel.
9. A data synchronization apparatus of a server, the data synchronization apparatus of the server comprising:
the receiving module is used for receiving file information sent by the client;
the acquisition module is used for acquiring a data channel corresponding to the file information;
the obtaining module is further configured to obtain a processing mode corresponding to the data channel, where the processing mode includes any one of update processing, insertion processing, deletion processing, and creation processing;
and the processing module is used for processing the file corresponding to the file information according to the processing mode.
10. A data synchronization apparatus of a server, characterized in that the data synchronization apparatus of the server comprises: memory, processor and a data synchronization program of a server stored on the memory and executable on the processor, the data synchronization program of the server when executed by the processor implementing the steps of the data synchronization method of a server according to any of claims 1 to 7.
11. A computer storage medium, characterized in that the computer storage medium has stored thereon a data synchronization program of a server, which when executed by a processor implements the steps of the data synchronization method of the server according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010445601.9A CN113704212B (en) | 2020-05-22 | 2020-05-22 | Data synchronization method, device and equipment of server and computer storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010445601.9A CN113704212B (en) | 2020-05-22 | 2020-05-22 | Data synchronization method, device and equipment of server and computer storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113704212A true CN113704212A (en) | 2021-11-26 |
| CN113704212B CN113704212B (en) | 2024-08-16 |
Family
ID=78646489
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010445601.9A Active CN113704212B (en) | 2020-05-22 | 2020-05-22 | Data synchronization method, device and equipment of server and computer storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113704212B (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070255766A1 (en) * | 2004-08-12 | 2007-11-01 | Telecom Italia S.P.A. | System, a Method and a Device for Updating a Data Set Through a Communication Network |
| CN103023809A (en) * | 2012-12-28 | 2013-04-03 | 中国船舶重工集团公司第七0九研究所 | Information system synchronous data processing method utilizing secondary buffer technology |
| US20130132340A1 (en) * | 2010-08-02 | 2013-05-23 | Beijing Lenovo Software Ltd. | File synchronization method, electronic device and synchronization system |
| CN103294704A (en) * | 2012-02-28 | 2013-09-11 | 鸿富锦精密工业（深圳）有限公司 | File synchronous system and method |
| US20170031948A1 (en) * | 2014-04-15 | 2017-02-02 | Huawei Technologies Co., Ltd. | File synchronization method, server, and terminal |
| CN107087038A (en) * | 2017-06-29 | 2017-08-22 | 珠海市魅族科技有限公司 | A kind of method of data syn-chronization, synchronizer, device and storage medium |
| CN108965383A (en) * | 2018-05-31 | 2018-12-07 | 康键信息技术（深圳）有限公司 | File synchronisation method, device, computer equipment and storage medium |
| CN110502583A (en) * | 2019-08-27 | 2019-11-26 | 深圳前海微众银行股份有限公司 | Distributed Data Synchronization method, apparatus, equipment and readable storage medium storing program for executing |
| WO2019228034A1 (en) * | 2018-05-30 | 2019-12-05 | 杭州海康威视数字技术股份有限公司 | Method and apparatus for data synchronization |
-
2020
- 2020-05-22 CN CN202010445601.9A patent/CN113704212B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070255766A1 (en) * | 2004-08-12 | 2007-11-01 | Telecom Italia S.P.A. | System, a Method and a Device for Updating a Data Set Through a Communication Network |
| US20130132340A1 (en) * | 2010-08-02 | 2013-05-23 | Beijing Lenovo Software Ltd. | File synchronization method, electronic device and synchronization system |
| CN103294704A (en) * | 2012-02-28 | 2013-09-11 | 鸿富锦精密工业（深圳）有限公司 | File synchronous system and method |
| CN103023809A (en) * | 2012-12-28 | 2013-04-03 | 中国船舶重工集团公司第七0九研究所 | Information system synchronous data processing method utilizing secondary buffer technology |
| US20170031948A1 (en) * | 2014-04-15 | 2017-02-02 | Huawei Technologies Co., Ltd. | File synchronization method, server, and terminal |
| CN107087038A (en) * | 2017-06-29 | 2017-08-22 | 珠海市魅族科技有限公司 | A kind of method of data syn-chronization, synchronizer, device and storage medium |
| WO2019228034A1 (en) * | 2018-05-30 | 2019-12-05 | 杭州海康威视数字技术股份有限公司 | Method and apparatus for data synchronization |
| CN108965383A (en) * | 2018-05-31 | 2018-12-07 | 康键信息技术（深圳）有限公司 | File synchronisation method, device, computer equipment and storage medium |
| CN110502583A (en) * | 2019-08-27 | 2019-11-26 | 深圳前海微众银行股份有限公司 | Distributed Data Synchronization method, apparatus, equipment and readable storage medium storing program for executing |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113704212B (en) | 2024-08-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9015316B2 (en) | Correlation of asynchronous business transactions | |
| US10268515B2 (en) | Releasing resources from processes in a particular order to provide required resources to start an application | |
| CN108804215B (en) | Task processing method and device and electronic equipment | |
| KR101865432B1 (en) | Message queue management | |
| CN112486629B (en) | Micro-service state detection method, micro-service state detection device, electronic equipment and storage medium | |
| CN112416581B (en) | Distributed calling system for timed tasks | |
| CN111324423B (en) | Method and device for monitoring processes in container, storage medium and computer equipment | |
| CN113407383B (en) | Main and standby system switching method and device, server and main and standby system | |
| CN110502366B (en) | Case execution method, device, equipment and computer readable storage medium | |
| CN110417586B (en) | Service monitoring method, service node, server and computer readable storage medium | |
| US20100306593A1 (en) | Automatic bug reporting tool | |
| CN110674034A (en) | Health examination method and device, electronic equipment and storage medium | |
| CN113238815A (en) | Interface access control method, device, equipment and storage medium | |
| CN111596993A (en) | Interface processing method, terminal equipment and storage medium | |
| EP2860633A1 (en) | Method for maintaining file system of computer system | |
| US9298539B1 (en) | Automated error recovery for workflows | |
| CN107729207B (en) | Application program main thread monitoring method and device | |
| CN107451056B (en) | Method and device for monitoring interface test result | |
| CN111342986A (en) | Distributed node management method and device, distributed system and storage medium | |
| CN113704212B (en) | Data synchronization method, device and equipment of server and computer storage medium | |
| CN112416735A (en) | Application program detection method and device, terminal equipment and storage medium | |
| US20160085638A1 (en) | Computer system and method of identifying a failure | |
| CN112596750B (en) | Application testing method and device, electronic equipment and computer readable storage medium | |
| CN114048094A (en) | Application start monitoring method, device, equipment and storage medium | |
| CN110362464B (en) | Software analysis method and equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |