CN114547195A - Batch data synchronization task processing method, device, equipment and storage medium - Google Patents

Batch data synchronization task processing method, device, equipment and storage medium Download PDF

Info

Publication number
CN114547195A
CN114547195A CN202210134854.3A CN202210134854A CN114547195A CN 114547195 A CN114547195 A CN 114547195A CN 202210134854 A CN202210134854 A CN 202210134854A CN 114547195 A CN114547195 A CN 114547195A
Authority
CN
China
Prior art keywords
target
data synchronization
synchronization task
partner
data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202210134854.3A
Other languages
Chinese (zh)
Other versions
CN114547195B (en
Inventor
马位涛
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ping An Technology Shenzhen Co Ltd
Original Assignee
Ping An Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ping An Technology Shenzhen Co Ltd filed Critical Ping An Technology Shenzhen Co Ltd
Priority to CN202210134854.3A priority Critical patent/CN114547195B/en
Publication of CN114547195A publication Critical patent/CN114547195A/en
Application granted granted Critical
Publication of CN114547195B publication Critical patent/CN114547195B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25—Integrating or interfacing systems involving database management systems
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes

Abstract

The application relates to the technical field of artificial intelligence, and discloses a batch data synchronization task processing method, a device, equipment and a storage medium, wherein the method comprises the following steps: splicing each file prefix in the partner interface configuration table with the current date by the file name to be processed; generating a data synchronization task to be added for each file name to be processed which is not contained in a preset partner data synchronization task table according to a partner interface configuration table, and adding the data synchronization task to be added into the partner data synchronization task table; capturing each data synchronization task of which the execution time window comprises the current time and the processing state is to be executed or failed from a partner data synchronization task table as a target data synchronization task; and according to the target data synchronization task, performing data synchronization between the target application system and a target partner system corresponding to the target data synchronization task. The docking period is shortened, the research and development cost is reduced, and the code redundancy is reduced.

Description

Batch data synchronization task processing method, device, equipment and storage medium
Technical Field
The present application relates to the field of artificial intelligence technologies, and in particular, to a method, an apparatus, a device, and a storage medium for processing batch data synchronization tasks.
Background
With the development of internet technology, multiple partner systems may be available for the same business process. If a plurality of partner systems need to be docked for data synchronization, an interface and a separate batch synchronization task need to be added when one partner system is added, docking development takes a long time, docking period is long, interface and batch task development cost is high, and code redundancy is caused because each partner system needs to develop a set of interface codes.
Disclosure of Invention
The application mainly aims to provide a batch data synchronization task processing method, a batch data synchronization task processing device, equipment and a storage medium, and aims to solve the technical problems of long docking period, high research and development cost and code redundancy in the prior art when a plurality of partner systems need to be docked for data synchronization in the same service flow.
In order to achieve the above object, the present application provides a batch data synchronization task processing method, including:
acquiring a task generation request corresponding to a target business process, wherein the task generation request carries a partner interface configuration table;
splicing each file prefix in the partner interface configuration table with the current date by the file name to be processed;
generating a data synchronization task to be added for each file name to be processed which is not contained in a preset data synchronization task table of the partner according to the interface configuration table of the partner, and adding the data synchronization task to be added into the data synchronization task table of the partner;
acquiring a data synchronization request corresponding to the target business process;
responding to the data synchronization request, and capturing each data synchronization task of which the execution time window comprises the current time and the processing state is to be executed or failed from the data synchronization task table of the partner as a target data synchronization task;
and according to the target data synchronization task, performing data synchronization between a target application system and a target partner system corresponding to the target data synchronization task.
Further, the step of generating a to-be-added data synchronization task for each to-be-processed file name that is not included in a preset partner data synchronization task table according to the partner interface configuration table includes:
acquiring one to-be-processed file name from each to-be-processed file name not included in the partner data synchronization task table as a target file name;
acquiring an execution time window, field selection configuration, synchronization type and address data configuration corresponding to the target file name from the partner interface configuration table to obtain task configuration data;
generating a data synchronization task according to the target file name and the task configuration data, wherein the data synchronization task is used as the data synchronization task to be added corresponding to the target file name;
and repeatedly executing the step of acquiring one to-be-processed file name from each to-be-processed file name not contained in the data synchronization task table of the partner as a target file name until the data synchronization task to be added corresponding to each to-be-processed file name not contained in the data synchronization task table of the partner is completed.
Further, the step of grabbing, from the partner data synchronization task table, each data synchronization task whose execution time window includes the current time and whose processing state is to be executed or failed as a target data synchronization task in response to the data synchronization request includes:
responding to the data synchronization request, and acquiring a node identification set needing synchronization corresponding to the target business process;
acquiring a timing synchronization task corresponding to each node identifier in the node identifier set to be synchronized from a timing task library as a target timing synchronization task;
executing each target timing synchronization task according to a target time interval corresponding to the target business process within a preset target synchronization time period;
and capturing each data synchronization task of which the execution time window comprises the current time and the processing state is to be executed or failed from the partner data synchronization task table as a target data synchronization task through the target timing synchronization task.
Further, the step of performing data synchronization between a target application system and a target partner system corresponding to the target data synchronization task according to the target data synchronization task includes:
if the synchronization type of the target data synchronization task is uploading, executing the target data synchronization task, and uploading data in the target application system to the target partner system;
and if the synchronization type of the target data synchronization task is downloading, executing the target data synchronization task, and updating the data in the target application system according to the data in the target partner system.
Further, the step of uploading the data in the target application system to the target partner system includes:
according to the field selection configuration of the target data synchronization task, the address data configuration and the starting time of an execution time window, acquiring data from the target application system to obtain data to be processed;
dividing files to be synchronized according to the data to be processed;
encrypting each file to be synchronized by adopting an MD5 encryption method to obtain a target hash value;
generating a directory table according to each file to be synchronized;
pushing each file to be synchronized, each target hash value and the directory table to a first storage space corresponding to an uploading address of the target data synchronization task in the target partner system to obtain a monitoring synchronization result;
and updating the processing state in the partner data synchronization task table according to the monitoring synchronization result.
Further, after the step of pushing each file to be synchronized, each target hash value, and the directory table to a first storage space corresponding to an upload address of the target data synchronization task in the target partner system to obtain a monitoring synchronization result, the method further includes:
obtaining a target reconciliation result sent by the target partner system, wherein the target partner system is configured to: according to the directory table in the first storage space, performing file quantity reconciliation and data number reconciliation on each file to be synchronized in the first storage space respectively to obtain a directory reconciliation result, performing content reconciliation on each file to be synchronized in the first storage space according to each target hash value in the first storage space to obtain a content reconciliation result, and taking the directory reconciliation result and the content reconciliation result as the target reconciliation result;
acquiring supplementary uploading configuration data sent by an auditing end according to the target account checking result;
generating an exception handling task according to the supplementary uploaded configuration data;
and executing the exception handling task, and uploading data corresponding to the supplementary uploading configuration data in the target application system to the target partner system.
Further, the step of updating the data in the target application system according to the data in the target partner system includes:
acquiring a file from a second storage space corresponding to the download address of the target data synchronization task in the target partner system to obtain a file to be updated;
storing the file to be updated in a third storage space corresponding to a local temporary storage address of the target data synchronization task in the target application system;
updating the data in the target application system according to the file to be updated in the third storage space to obtain an update ending signal;
and deleting the file to be updated in the third storage space based on the update ending signal.
The present application further proposes a batch data synchronization task processing apparatus, the apparatus comprising:
the system comprises a first request acquisition module, a first service processing module and a second service processing module, wherein the first request acquisition module is used for acquiring a task generation request corresponding to a target service process, and the task generation request carries a partner interface configuration table;
the file name splicing module to be processed is used for splicing each file prefix in the partner interface configuration table with the current date;
the data synchronization task list updating module of the partner is used for generating a data synchronization task to be added for each file name to be processed which is not contained in a preset data synchronization task list of the partner according to the interface configuration list of the partner and adding the data synchronization task to be added into the data synchronization task list of the partner;
the second request acquisition module is used for acquiring a data synchronization request corresponding to the target business process;
a target data synchronization task determination module, configured to respond to the data synchronization request, and capture, from the partner data synchronization task table, each data synchronization task whose execution time window includes current time and whose processing state is to be executed or failed, as a target data synchronization task;
and the data synchronization module is used for carrying out data synchronization between a target application system and a target partner system corresponding to the target data synchronization task according to the target data synchronization task.
The present application further proposes a computer device comprising a memory and a processor, the memory storing a computer program, the processor implementing the steps of any of the above methods when executing the computer program.
The present application also proposes a computer-readable storage medium having stored thereon a computer program which, when being executed by a processor, carries out the steps of the method of any of the above.
The method comprises the steps of splicing each file prefix in a partner interface configuration table with a current date by a file name to be processed; generating a data synchronization task to be added for each file name to be processed which is not contained in a preset data synchronization task table of the partner according to the interface configuration table of the partner, and adding the data synchronization task to be added into the data synchronization task table of the partner; capturing each data synchronization task of which the execution time window comprises the current time and the processing state is to be executed or failed from the partner data synchronization task table as a target data synchronization task; and according to the target data synchronization task, performing data synchronization between a target application system and a target partner system corresponding to the target data synchronization task. The file names to be processed are spliced through each file prefix in the partner interface configuration table and the current date, and the data synchronization tasks are automatically generated according to the file names to be processed, so that the data synchronization tasks of each partner system in the target service process are generated in real time based on configuration, an interface and an independent batch synchronization task do not need to be newly added for each partner system, the docking period is shortened, the research and development cost is reduced, and the code redundancy is reduced.
Drawings
FIG. 1 is a schematic flow chart illustrating a batch data synchronization task processing method according to an embodiment of the present disclosure;
FIG. 2 is a block diagram illustrating a batch data synchronization task processing apparatus according to an embodiment of the present disclosure;
fig. 3 is a block diagram illustrating a structure of a computer device according to an embodiment of the present application.
The implementation, functional features and advantages of the objectives of the present application will be further explained with reference to the accompanying drawings.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
Referring to fig. 1, an embodiment of the present application provides a batch data synchronization task processing method, where the method includes:
s1: acquiring a task generation request corresponding to a target business process, wherein the task generation request carries a partner interface configuration table;
s2: splicing the file prefixes in the partner interface configuration table with the current date by the file names to be processed;
s3: generating a data synchronization task to be added for each file name to be processed which is not contained in a preset data synchronization task table of the partner according to the interface configuration table of the partner, and adding the data synchronization task to be added into the data synchronization task table of the partner;
s4: acquiring a data synchronization request corresponding to the target business process;
s5: responding to the data synchronization request, and capturing each data synchronization task of which the execution time window comprises the current time and the processing state is to be executed or failed from the data synchronization task table of the partner as a target data synchronization task;
s6: and according to the target data synchronization task, performing data synchronization between a target application system and a target partner system corresponding to the target data synchronization task.
According to the method and the device, the file names to be processed are spliced through the file prefixes and the current date in the partner interface configuration table, the data synchronization task is automatically generated according to the file names to be processed, the data synchronization task of each partner system in the target service process is generated in real time based on configuration, an interface and an independent batch synchronization task do not need to be newly added for each partner system, the docking period is shortened, the research and development cost is reduced, and code redundancy is reduced.
For S1, a task generation request corresponding to the target business process input by the user may be obtained, a task generation request corresponding to the target business process sent by the third-party application system may also be obtained, or a task generation request corresponding to the target business process actively generated by the program according to the preset conditions.
Optionally, a preset cycle is adopted to execute a task, and the task generation request is generated for the target business process according to a preset time interval. Therefore, the program of the application actively generates the task generation request corresponding to the target business process according to the preset conditions.
Alternatively, the preset time interval is set to 1 point per day.
The target business process is a business process needing batch data synchronization. The business process comprises a plurality of nodes, and each node in the business process which needs data synchronization corresponds to one synchronization type and one node identification. The synchronization types include: uploading or downloading.
The task generation request is a request for generating a data synchronization task.
The partner interface configuration table is a list for configuring an interface for each partner aiming at each node needing data synchronization in the target business process.
The partner interface configuration table includes: the method comprises the steps of flow identification, partner identification, node identification, file prefix, execution time window, field selection configuration, synchronization type and address data configuration. When the synchronization type is uploading, the address data configuration comprises the following steps: and uploading the address. When the synchronization type is downloading, the address data configuration comprises: download address, local temporary storage address. The field selection configuration is configuration data of the fields that need to be synchronized. The flow identifier may be a flow name, a flow ID, or the like, which uniquely identifies a flow. The partner identification may be a partner name, a partner ID, or the like, that uniquely identifies a partner. The node identification may be a node name, a node ID, or the like, which uniquely identifies a node. The file prefix is a string. The execution time window includes: a start time and an end time.
It is understood that the product of the number of nodes requiring data synchronization and the number of partners of the target business process is equal to the number of configuration pieces in the partner interface configuration table. The partner identifier and the node identifier are used as associated data, and the number of the associated data in the partner interface configuration table is the configuration number.
Optionally, the file prefix directly adopts a node identifier.
Optionally, the step of obtaining a task generation request corresponding to a target business process includes: acquiring a configuration request; responding to the configuration request, and acquiring configuration data to be stored corresponding to the target business process; updating the partner interface configuration table according to the configuration data to be stored; and acquiring the task generation request corresponding to the target business process according to the updated partner interface configuration table.
And S2, sequentially splicing each file prefix in the partner interface configuration table with the current date, and taking each character string obtained by splicing as a file name to be processed. That is, the number of file names to be processed is the same as the number of configuration pieces in the partner interface configuration table. And sequentially splicing, namely splicing the current date behind the file prefix.
For S3, the to-be-processed filename that is not included in the preset partner data synchronization task table is a filename for which a data synchronization task has not been determined, so according to the partner interface configuration table, a to-be-added data synchronization task is generated for each to-be-processed filename that is not included in the preset partner data synchronization task table, where parameters of the to-be-added data synchronization task are determined according to parameters (i.e., an execution time window, a field selection configuration, a synchronization type, and an address data configuration) of the partner interface configuration table; and adding the data synchronization task to be added into the partner data synchronization task table, and setting the processing state corresponding to the data synchronization task to be added in the partner data synchronization task table to be executed, so as to provide a basis for the subsequent automatic data synchronization.
The partner data synchronization task table comprises: file name, execution time window, field selection configuration, synchronization type, address data configuration, and processing state.
For S4, a data synchronization request corresponding to the target business process input by the user may be acquired, a data synchronization request corresponding to the target business process sent by a third-party application system may also be acquired, and a data synchronization request corresponding to the target business process actively generated by the program according to a preset condition may also be implemented.
Optionally, judging whether the current time is equal to the start time of the target synchronization time period corresponding to the target service process; and if so, generating the data synchronization request aiming at the target business process. Therefore, the program actively generates the data synchronization request corresponding to the target business process according to the preset conditions.
The data synchronization request is a request for data synchronization.
For S5, when the data synchronization request is received, it starts to fetch each data synchronization task whose execution time window includes the current time and whose processing status is to be executed or failed from the data synchronization task table of the partner, and takes each fetched data synchronization task as a target data synchronization task, that is, the time (that is, the current time) for fetching the target data synchronization task is located in the execution time window of the target data synchronization task, and the processing status of the target data synchronization task is to be executed or failed.
And S6, according to the field selection configuration, the synchronization type and the address data configuration carried by the target data synchronization task, carrying out data synchronization between the target application system and the target partner system corresponding to the target data synchronization task.
The target application system is an application system for realizing a target business process.
In an embodiment, the step of generating a to-be-added data synchronization task for each to-be-processed file name that is not included in a preset partner data synchronization task table according to the partner interface configuration table includes:
s31: acquiring one to-be-processed file name from each to-be-processed file name not included in the partner data synchronization task table as a target file name;
s32: acquiring an execution time window, field selection configuration, synchronization type and address data configuration corresponding to the target file name from the partner interface configuration table to obtain task configuration data;
s33: generating a data synchronization task according to the target file name and the task configuration data, wherein the data synchronization task is used as the data synchronization task to be added corresponding to the target file name;
s34: and repeatedly executing the step of acquiring one to-be-processed file name from each to-be-processed file name not contained in the data synchronization task table of the partner as a target file name until the data synchronization task to be added corresponding to each to-be-processed file name not contained in the data synchronization task table of the partner is completed.
According to the embodiment, the data synchronization task to be added is generated according to the execution time window, the field selection configuration, the synchronization type, the address data configuration and the file name to be processed, so that a foundation is provided for carrying out automatic data synchronization based on the data synchronization task to be added.
For S31, one of the to-be-processed filenames is sequentially obtained from each of the to-be-processed filenames that are not included in the partner data synchronization task table, and the obtained to-be-processed filename is taken as a target filename.
For S32, an execution time window, a field selection configuration, a synchronization type, and an address data configuration corresponding to the file prefix of the target file name are acquired from the partner interface configuration table, and the acquired execution time window, field selection configuration, synchronization type, and address data configuration are used as task configuration data.
For step S33, a data synchronization task is generated according to the target file name, and a parameter of the data synchronization task is set according to the task configuration data, and the set data synchronization task is used as the data synchronization task to be added corresponding to the target file name.
For S34, the steps S31 to S34 are repeatedly executed until the data synchronization task to be added corresponding to each of the to-be-processed file names determined not to be included in the partner data synchronization task table is completed.
In an embodiment, the step of fetching, from the data synchronization task table of the partner in response to the data synchronization request, each data synchronization task whose execution time window includes the current time and whose processing state is to be executed or failed as a target data synchronization task includes:
s51: responding to the data synchronization request, and acquiring a node identification set needing synchronization corresponding to the target business process;
s52: acquiring a timing synchronization task corresponding to each node identifier in the node identifier set to be synchronized from a timing task library as a target timing synchronization task;
s53: executing each target timing synchronization task according to a target time interval corresponding to the target business process within a preset target synchronization time period;
s54: and capturing each data synchronization task of which the execution time window comprises the current time and the processing state is to be executed or failed from the partner data synchronization task table as a target data synchronization task through the target timing synchronization task.
In this embodiment, each target timing synchronization task is executed according to a target time interval corresponding to the target business process within a preset target synchronization time period, and each data synchronization task whose execution time window includes current time and whose processing state is to be executed or failed is captured from the data synchronization task table of the partner through the target timing synchronization task, so that data synchronization is performed by automatically traversing the data synchronization task table of the partner, and synchronization efficiency is improved.
For S51, when receiving the data synchronization request, the node identifiers that need to be synchronized and correspond to the target business process may be obtained from a database as the node identifier sets that need to be synchronized and correspond to the target business process, or the node identifier sets that need to be synchronized and correspond to the target business process may be written into a program that implements the present application.
For S52, a timing synchronization task corresponding to each node identifier in the node identifier set to be synchronized is obtained from the timing task library as a target timing synchronization task, that is, each node identifier corresponds to one target timing synchronization task, and the number of the target timing synchronization tasks is the same as the number of each node identifier corresponding to the target service process and needing to be synchronized.
For S53, each of the target timing synchronization tasks is executed at a target time interval corresponding to the target business process within a preset target synchronization time period, that is, each of the target timing synchronization tasks is executed within the target synchronization time period with the start time of the target synchronization time period as a start, with the end time of the target synchronization time period as an end, and with the target time interval as a cycle.
Parameters carried by the target timing synchronization task include: and identifying the node.
For S54, capturing each data synchronization task whose execution time window includes the current time and whose processing state is to be executed or failed from the partner data synchronization task table through the target timing synchronization task, and regarding each captured data synchronization task as a target data synchronization task.
In an embodiment, the step of performing data synchronization between the target application system and the target partner system corresponding to the target data synchronization task according to the target data synchronization task includes:
s61: if the synchronization type of the target data synchronization task is uploading, executing the target data synchronization task, and uploading data in the target application system to the target partner system;
s62: and if the synchronization type of the target data synchronization task is downloading, executing the target data synchronization task, and updating the data in the target application system according to the data in the target partner system.
According to the embodiment, data synchronization is carried out between the target application system and the target partner system corresponding to the target data synchronization task according to the synchronization type, so that the personalized node data synchronization requirement is met.
For S61, if the synchronization type of the target data synchronization task is upload, it means that the data in the target application system needs to be uploaded to the target partner system, so the target data synchronization task is executed, the data is extracted from the target application system, and the extracted data is uploaded to the target partner system.
For S62, if the synchronization type of the target data synchronization task is downloading, it means that the data in the target application system needs to be updated according to the data in the target partner system, so the target data synchronization task is executed, the data is downloaded from the target partner system, and the data in the service table in the target application system is updated according to the downloaded data.
In an embodiment, the step of uploading the data in the target application system to the target partner system includes:
s611: according to the field selection configuration of the target data synchronization task, the address data configuration and the starting time of an execution time window, acquiring data from the target application system to obtain data to be processed;
s612: dividing files to be synchronized according to the data to be processed;
s613: encrypting each file to be synchronized by adopting an MD5 encryption method to obtain a target hash value;
s614: generating a directory table according to each file to be synchronized;
s615: pushing each file to be synchronized, each target hash value and the directory table to a first storage space corresponding to an uploading address of the target data synchronization task in the target partner system to obtain a monitoring synchronization result;
s616: and updating the processing state in the partner data synchronization task table according to the monitoring synchronization result.
According to the embodiment, the division of the files to be synchronized of the data to be processed is realized, the target hash value of the files to be synchronized is generated, the directory table is generated according to the files to be synchronized, the target hash value and the directory table are uploaded and the monitoring synchronization result is generated, finally, the processing state in the partner data synchronization task table is updated according to the monitoring synchronization result, the data synchronization, the monitoring of the data synchronization result and the processing of the monitoring result are automatically performed, the closed loop of the data uploading is realized, and the accuracy of the data uploading is improved.
For S611, configuration and address data configuration are selected according to the field of the target data synchronization task, data in the previous data cycle is obtained from the target application system, and the obtained data is used as data to be processed.
The time period between the start time of the current execution time window and the start time of the last execution time window adjacent to the current execution time window is taken as the last data cycle.
And S612, dividing the to-be-synchronized file of the to-be-processed data by adopting a preset data number threshold. That is, instead of the number of pieces of data of the last divided file to be synchronized being the same as the number of pieces of data threshold, the number of pieces of data of the last divided file to be synchronized is less than or equal to the number of pieces of data threshold.
For step S613, each file to be synchronized is encrypted by using an MD5 (information digest algorithm) encryption method, and the encrypted hash value is used as a target hash value.
And S614, extracting data from each file to be synchronized to generate a directory table.
The directory table includes: file identification and number of data pieces. The file identifier may be a file name, a file ID, or the like, which uniquely identifies a copy of the data of the file to be synchronized.
For S615, putting each target hash value into the same list to obtain a hash value list; and pushing each file to be synchronized, the hash value list and the directory list to a first storage space corresponding to an uploading address of the target data synchronization task in the target partner system, and generating a monitoring synchronization result according to the monitored pushing result.
The monitoring synchronization result comprises: key word and synchronous result, the value range of key word includes: file identification, directory table identification, and hash value list identification. The synchronization result includes: success or failure. The directory table identification may be a directory table name, a directory table ID, or the like, that uniquely identifies a directory table. The hash value list identifier may be a hash value list name, a hash value list ID, or the like, which uniquely identifies a hash value list.
For S616, the keyword whose synchronization result in the monitoring synchronization result is failure is used as the keyword to be updated; and updating the processing state corresponding to the keyword to be updated in the partner data synchronization task table to be failure.
In an embodiment, after the step of pushing each of the files to be synchronized, each of the target hash values, and the directory table to the first storage space corresponding to the upload address of the target data synchronization task in the target partner system to obtain the monitoring synchronization result, the method further includes:
s617: obtaining a target reconciliation result sent by the target partner system, wherein the target partner system is configured to: according to the directory table in the first storage space, performing file quantity reconciliation and data number reconciliation on each file to be synchronized in the first storage space respectively to obtain a directory reconciliation result, performing content reconciliation on each file to be synchronized in the first storage space according to each target hash value in the first storage space to obtain a content reconciliation result, and taking the directory reconciliation result and the content reconciliation result as the target reconciliation result;
s618: acquiring supplementary uploading configuration data sent by an auditing end according to the target account checking result;
s619: generating an exception handling task according to the supplementary uploaded configuration data;
s6110: and executing the exception handling task, and uploading data corresponding to the supplementary uploading configuration data in the target application system to the target partner system.
In the embodiment, the target account checking result sent by the target partner system is obtained, the supplementary uploading configuration data sent by the auditing end is obtained according to the target account checking result, and finally the data corresponding to the supplementary uploading configuration data in the target application system is uploaded to the target partner system, so that the data account checking is carried out on the receiving end, and the data is prevented from being modified in the transmission process to influence the accuracy of data synchronization.
For S617, determining whether a specified file identifier exists in the first storage space, where the specified file identifier is any file identifier in the directory table in the first storage space; if yes, determining that the reconciliation result of the file identifier corresponding to the specified file identifier is passed, otherwise determining that the reconciliation result of the file identifier corresponding to the specified file identifier is not passed; judging whether the number of data pieces corresponding to the specified file identification in the directory table in the first storage space is equal to the number of data pieces of the file to be synchronized corresponding to the specified file identification in the first storage space; if the two items are equal, determining that the number of reconciliation rules corresponding to the specified file identification pass, otherwise, determining that the number of reconciliation rules corresponding to the specified file identification do not pass; and taking the account checking result of each file identification and the account checking rule of each number as an inventory account checking result.
Encrypting each file to be synchronized in the first storage space by using an MD5 encryption method to obtain hash values to be compared; and when the target hash value corresponding to the specified file identification in the first storage space is the same as the hash value to be compared corresponding to the specified file identification, determining that the hash value comparison result corresponding to the specified file identification passes, otherwise, determining that the hash value comparison result corresponding to the specified file identification does not pass, and taking each hash value comparison result as a content reconciliation result.
For the step S618, the target reconciliation result is sent to the auditing end through the communication connection with the auditing end; the user audits the target account checking result at the auditing end and inputs supplement uploading configuration data; and acquiring supplementary uploading configuration data which is sent by the auditing end and corresponds to the target reconciliation result.
And for the step S619, generating an exception handling task according to each piece of supplementary uploading configuration in the supplementary uploading configuration data.
The supplemental upload configuration includes: file identification, execution time window, field selection configuration, synchronization type and address data configuration.
For S6110, the exception handling task is executed, and the data corresponding to the supplementary upload configuration data in the target application system is uploaded to the target partner system, thereby implementing supplementary upload.
In an embodiment, the step of updating the data in the target application system according to the data in the target partner system includes:
s621: acquiring a file from a second storage space corresponding to the download address of the target data synchronization task in the target partner system to obtain a file to be updated;
s622: storing the file to be updated in a third storage space corresponding to a local temporary storage address of the target data synchronization task in the target application system;
s623: updating the data in the target application system according to the file to be updated in the third storage space to obtain an update ending signal;
s624: and deleting the file to be updated in the third storage space based on the update ending signal.
According to the embodiment, the files in the next week of the target partner system are cached in the third storage space, then the files are obtained from the third storage space to update the data in the target application system, and the files in the third storage space are deleted after updating is completed, so that the accuracy of updating the data in the target application system is improved, the files temporarily cached in the third storage space are deleted in time, and the storage space is saved.
For step S621, a file is obtained from the second storage space corresponding to the download address of the target data synchronization task in the target partner system, and the obtained file is used as a file to be updated. And the file to be updated is data corresponding to the execution time window and the field selection configuration of the target data synchronization task.
For S622, the file to be updated is stored in the third storage space corresponding to the local temporary storage address of the target data synchronization task in the target application system, so that file caching is implemented.
For step S623, capturing detail data from the file to be updated in the third storage space, updating the service table in the target application system with the captured detail data, and generating an update end signal when all the detail data in the file to be updated is updated in the service table in the target application system.
For S624, in response to the update end signal, the file to be updated in the third storage space is deleted, so that the storage space is saved.
Referring to fig. 2, the present application also proposes a batch data synchronization task processing apparatus, including:
a first request obtaining module 100, configured to obtain a task generation request corresponding to a target business process, where the task generation request carries a partner interface configuration table;
a pending filename concatenation module 200, configured to concatenate each file prefix in the partner interface configuration table with a current date to be processed;
a partner data synchronization task table updating module 300, configured to generate a data synchronization task to be added for each to-be-processed file name that is not included in a preset partner data synchronization task table according to the partner interface configuration table, and add the data synchronization task to be added to the partner data synchronization task table;
a second request obtaining module 400, configured to obtain a data synchronization request corresponding to the target business process;
a target data synchronization task determining module 500, configured to respond to the data synchronization request, and capture, from the partner data synchronization task table, each data synchronization task whose execution time window includes current time and whose processing state is to be executed or failed, as a target data synchronization task;
and a data synchronization module 600, configured to perform data synchronization between the target application system and the target partner system corresponding to the target data synchronization task according to the target data synchronization task.
According to the method and the device, the file names to be processed are spliced through the file prefixes and the current date in the partner interface configuration table, the data synchronization task is automatically generated according to the file names to be processed, the data synchronization task of each partner system in the target service process is generated in real time based on configuration, an interface and an independent batch synchronization task do not need to be newly added for each partner system, the docking period is shortened, the research and development cost is reduced, and code redundancy is reduced.
Referring to fig. 3, a computer device, which may be a server and whose internal structure may be as shown in fig. 3, is also provided in the embodiment of the present application. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the computer designed processor is used to provide computational and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer equipment is used for storing data such as batch data synchronization task processing methods and the like. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a batch data synchronization task processing method. The batch data synchronization task processing method comprises the following steps: acquiring a task generation request corresponding to a target business process, wherein the task generation request carries a partner interface configuration table; splicing each file prefix in the partner interface configuration table with the current date by the file name to be processed; generating a data synchronization task to be added for each file name to be processed which is not contained in a preset data synchronization task table of the partner according to the interface configuration table of the partner, and adding the data synchronization task to be added into the data synchronization task table of the partner; acquiring a data synchronization request corresponding to the target business process; responding to the data synchronization request, and capturing each data synchronization task of which the execution time window comprises the current time and the processing state is to be executed or failed from the data synchronization task table of the partner as a target data synchronization task; and according to the target data synchronization task, performing data synchronization between a target application system and a target partner system corresponding to the target data synchronization task.
According to the method and the device, the file names to be processed are spliced through the file prefixes and the current date in the partner interface configuration table, the data synchronization task is automatically generated according to the file names to be processed, the data synchronization task of each partner system in the target service process is generated in real time based on configuration, an interface and an independent batch synchronization task do not need to be newly added for each partner system, the docking period is shortened, the research and development cost is reduced, and code redundancy is reduced.
An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements a batch data synchronization task processing method, including the steps of: acquiring a task generation request corresponding to a target business process, wherein the task generation request carries a partner interface configuration table; splicing each file prefix in the partner interface configuration table with the current date by the file name to be processed; generating a data synchronization task to be added for each file name to be processed which is not contained in a preset data synchronization task table of the partner according to the interface configuration table of the partner, and adding the data synchronization task to be added into the data synchronization task table of the partner; acquiring a data synchronization request corresponding to the target business process; responding to the data synchronization request, and capturing each data synchronization task of which the execution time window comprises the current time and the processing state is to be executed or failed from the data synchronization task table of the partner as a target data synchronization task; and according to the target data synchronization task, performing data synchronization between a target application system and a target partner system corresponding to the target data synchronization task.
According to the executed batch data synchronization task processing method, the file names to be processed are spliced through the file prefixes and the current date in the interface configuration table of the cooperative party, the data synchronization task is automatically generated according to the file names to be processed, the data synchronization task of each cooperative party system in the target service process is generated in real time based on configuration, an interface and an independent batch synchronization task do not need to be newly added for each cooperative party system, the docking period is shortened, the research and development cost is reduced, and the code redundancy is reduced.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above may be implemented by hardware instructions of a computer program, which may be stored in a non-volatile computer-readable storage medium, and when executed, may include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium provided herein and used in the examples may include non-volatile and/or volatile memory. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), double-rate SDRAM (SSRSDRAM), Enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and bus dynamic RAM (RDRAM).
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, apparatus, article, or method. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, apparatus, article, or method that includes the element.
The above description is only a preferred embodiment of the present application, and not intended to limit the scope of the present application, and all modifications of equivalent structures and equivalent processes, which are made by the contents of the specification and the drawings of the present application, or which are directly or indirectly applied to other related technical fields, are also included in the scope of the present application.

Claims (10)

1. A batch data synchronization task processing method is characterized by comprising the following steps:
acquiring a task generation request corresponding to a target business process, wherein the task generation request carries a partner interface configuration table;
splicing each file prefix in the partner interface configuration table with the current date by the file name to be processed;
generating a data synchronization task to be added for each file name to be processed which is not contained in a preset data synchronization task table of the partner according to the interface configuration table of the partner, and adding the data synchronization task to be added into the data synchronization task table of the partner;
acquiring a data synchronization request corresponding to the target business process;
responding to the data synchronization request, and capturing each data synchronization task of which the execution time window comprises the current time and the processing state is to be executed or failed from the data synchronization task table of the partner as a target data synchronization task;
and according to the target data synchronization task, performing data synchronization between a target application system and a target partner system corresponding to the target data synchronization task.
2. The batch data synchronization task processing method according to claim 1, wherein the step of generating a synchronization task of data to be added for each to-be-processed file name that is not included in a preset partner data synchronization task table according to the partner interface configuration table includes:
acquiring one to-be-processed file name from each to-be-processed file name not included in the partner data synchronization task table as a target file name;
acquiring an execution time window, field selection configuration, synchronization type and address data configuration corresponding to the target file name from the partner interface configuration table to obtain task configuration data;
generating a data synchronization task according to the target file name and the task configuration data, wherein the data synchronization task is used as the data synchronization task to be added corresponding to the target file name;
and repeatedly executing the step of acquiring one to-be-processed file name from each to-be-processed file name not contained in the data synchronization task table of the partner as a target file name until the data synchronization task to be added corresponding to each to-be-processed file name not contained in the data synchronization task table of the partner is completed.
3. The batch data synchronization task processing method according to claim 1, wherein the step of grabbing, as a target data synchronization task, each data synchronization task whose execution time window contains a current time and whose processing state is to be executed or failed from the partner data synchronization task table in response to the data synchronization request includes:
responding to the data synchronization request, and acquiring a node identification set needing synchronization corresponding to the target business process;
acquiring a timing synchronization task corresponding to each node identifier in the node identifier set to be synchronized from a timing task library as a target timing synchronization task;
executing each target timing synchronization task according to a target time interval corresponding to the target business process within a preset target synchronization time period;
and capturing each data synchronization task of which the execution time window comprises the current time and the processing state is to be executed or failed from the partner data synchronization task table as a target data synchronization task through the target timing synchronization task.
4. The batch data synchronization task processing method according to claim 1, wherein the step of performing data synchronization between a target application system and a target partner system corresponding to the target data synchronization task according to the target data synchronization task includes:
if the synchronization type of the target data synchronization task is uploading, executing the target data synchronization task, and uploading data in the target application system to the target partner system;
and if the synchronization type of the target data synchronization task is downloading, executing the target data synchronization task, and updating the data in the target application system according to the data in the target partner system.
5. The batch data synchronization task processing method of claim 4, wherein the step of uploading the data in the target application system to the target partner system comprises:
according to the field selection configuration of the target data synchronization task, the address data configuration and the starting time of an execution time window, acquiring data from the target application system to obtain data to be processed;
dividing files to be synchronized according to the data to be processed;
encrypting each file to be synchronized by adopting an MD5 encryption method to obtain a target hash value;
generating a directory table according to each file to be synchronized;
pushing each file to be synchronized, each target hash value and the directory table to a first storage space corresponding to an uploading address of the target data synchronization task in the target partner system to obtain a monitoring synchronization result;
and updating the processing state in the partner data synchronization task table according to the monitoring synchronization result.
6. The batch data synchronization task processing method according to claim 5, wherein after the step of pushing each file to be synchronized, each target hash value, and the directory table to a first storage space in the target partner system corresponding to an upload address of the target data synchronization task to obtain a monitoring synchronization result, the method further comprises:
obtaining a target reconciliation result sent by the target partner system, wherein the target partner system is configured to: according to the directory table in the first storage space, performing file quantity reconciliation and data quantity reconciliation on each file to be synchronized in the first storage space respectively to obtain a directory reconciliation result, performing content reconciliation on each file to be synchronized in the first storage space according to each target hash value in the first storage space to obtain a content reconciliation result, and taking the directory reconciliation result and the content reconciliation result as the target reconciliation result;
acquiring supplementary uploading configuration data sent by an auditing end according to the target account checking result;
generating an exception handling task according to the supplementary uploaded configuration data;
and executing the exception handling task, and uploading data corresponding to the supplementary uploading configuration data in the target application system to the target partner system.
7. The batch data synchronization task processing method according to claim 4, wherein the step of updating the data in the target application system according to the data in the target partner system includes:
acquiring a file from a second storage space corresponding to the download address of the target data synchronization task in the target partner system to obtain a file to be updated;
storing the file to be updated in a third storage space corresponding to a local temporary storage address of the target data synchronization task in the target application system;
updating the data in the target application system according to the file to be updated in the third storage space to obtain an update ending signal;
and deleting the file to be updated in the third storage space based on the update ending signal.
8. A batch data synchronization task processing apparatus, characterized in that the apparatus comprises:
the system comprises a first request acquisition module, a first service processing module and a second service processing module, wherein the first request acquisition module is used for acquiring a task generation request corresponding to a target service process, and the task generation request carries a partner interface configuration table;
the file name splicing module to be processed is used for splicing each file prefix in the partner interface configuration table with the current date;
the data synchronization task list updating module of the partner is used for generating a data synchronization task to be added for each file name to be processed which is not contained in a preset data synchronization task list of the partner according to the interface configuration list of the partner and adding the data synchronization task to be added into the data synchronization task list of the partner;
the second request acquisition module is used for acquiring a data synchronization request corresponding to the target business process;
a target data synchronization task determination module, configured to respond to the data synchronization request, and capture, from the partner data synchronization task table, each data synchronization task whose execution time window includes current time and whose processing state is to be executed or failed, as a target data synchronization task;
and the data synchronization module is used for carrying out data synchronization between a target application system and a target partner system corresponding to the target data synchronization task according to the target data synchronization task.
9. A computer device comprising a memory and a processor, the memory storing a computer program, wherein the processor implements the steps of the method of any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.
CN202210134854.3A 2022-02-14 2022-02-14 Batch data synchronous task processing method, device, equipment and storage medium Active CN114547195B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210134854.3A CN114547195B (en) 2022-02-14 2022-02-14 Batch data synchronous task processing method, device, equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210134854.3A CN114547195B (en) 2022-02-14 2022-02-14 Batch data synchronous task processing method, device, equipment and storage medium

Publications (2)

Publication Number Publication Date
CN114547195A true CN114547195A (en) 2022-05-27
CN114547195B CN114547195B (en) 2024-05-07

Family

ID=81676069

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210134854.3A Active CN114547195B (en) 2022-02-14 2022-02-14 Batch data synchronous task processing method, device, equipment and storage medium

Country Status (1)

Country Link
CN (1) CN114547195B (en)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180365265A1 (en) * 2017-06-16 2018-12-20 Thales Synchronization of a dual avionic and non-avionic system
CN110543448A (en) * 2019-07-22 2019-12-06 深圳壹账通智能科技有限公司 data synchronization method, device, equipment and computer readable storage medium
CN111654522A (en) * 2020-04-27 2020-09-11 平安证券股份有限公司 File synchronization method, file synchronization server and storage medium

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180365265A1 (en) * 2017-06-16 2018-12-20 Thales Synchronization of a dual avionic and non-avionic system
CN109150954A (en) * 2017-06-16 2019-01-04 泰勒斯公司 Aviation electronics is synchronous with non-aviation electronics dual system
CN110543448A (en) * 2019-07-22 2019-12-06 深圳壹账通智能科技有限公司 data synchronization method, device, equipment and computer readable storage medium
CN111654522A (en) * 2020-04-27 2020-09-11 平安证券股份有限公司 File synchronization method, file synchronization server and storage medium

Also Published As

Publication number Publication date
CN114547195B (en) 2024-05-07

Similar Documents

Publication Publication Date Title
CN108520059A (en) Loading method, system, computer equipment and the storage medium of the client H5 pages
CN104699511A (en) Plug-in upgrading method and plug-in upgrading device
CN108345467B (en) Version tree file updating method and device
CN111263354A (en) OTA differential upgrading method and device
CN110597518A (en) Project construction method and device, computer equipment and storage medium
CN111198709A (en) RPM software package management method, system, device and readable storage medium
CN104932887A (en) Plug-in management method of platform system and platform system
CN111177121A (en) Order data feedback method and device, computer equipment and storage medium
CN109101232B (en) Method, device, computer equipment and storage medium for product development
CN113961179A (en) Service access method, system, electronic device and storage medium of SOAR platform
CN114547195A (en) Batch data synchronization task processing method, device, equipment and storage medium
CN113779513A (en) ZIP file identification insertion method and device, computer equipment and storage medium
CN113439424B (en) Establishing a blockchain having blocks including an adjustable number of transaction blocks and a plurality of intermediate blocks
CN108460116B (en) Search method, search device, computer equipment, storage medium and search system
CN113489650B (en) Routing method, device and equipment of client and storage medium
CN113472915B (en) Domain name resolution method, device, equipment and storage medium
CN110569240A (en) Data storage method and device, computer equipment and storage medium
CN115640471A (en) Short link generation method and device, computer equipment and storage medium
CN111400243B (en) Development management system based on pipeline service and file storage method and device
CN113282300A (en) Volume optimization method, device, equipment and storage medium for application installation package
CN113486267A (en) Analysis method, device and equipment of application entry page and storage medium
CN114968963A (en) File overwriting method and device and electronic equipment
CN112800181A (en) Text retrieval method and device, computer equipment and storage medium
CN111930411A (en) Code upgrading method and device, computer equipment and storage medium
CN113721968B (en) Firmware upgrading method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant