WO2024067832A1

WO2024067832A1 - Data processing method and apparatus, electronic device, and storage medium

Info

Publication number: WO2024067832A1
Application number: PCT/CN2023/122758
Authority: WO
Inventors: 张鑫; 何卓航
Original assignee: 顺丰科技有限公司
Priority date: 2022-09-29
Filing date: 2023-09-28
Publication date: 2024-04-04
Also published as: CN117827854A

Abstract

The present application provides a data processing method and apparatus, an electronic device, and a storage medium. The method comprises: sending data update notification information to a peripheral system; if it is determined that the peripheral system has received all data to be processed in a set of data to be processed, sending data take-effect notification information to the peripheral system; and if it is determined that the peripheral system has enabled all the data to be processed in the set of data to be processed to take effect, updating a data processing task state corresponding to the set of data to be processed to have taken effect. According to the present application, data processing of a large data volume is facilitated, and after it is determined that all the data to be processed has been received by the peripheral system, the task is allowed to take effect, so that it is ensured that the atomicity of a large batch of different types of data takes effect, and the correctness, durability and integrity of the data are ensured; data version management is realized, and the durability and integrity of the data are ensured.

Description

Data processing method, device, electronic device and storage medium

Technical Field

The present application relates to the field of computer technology, and in particular to a data processing method, device, electronic device and storage medium.

Background of the Invention

With the development of technology, people's daily lives have gradually begun to use various systems to improve production efficiency. During the system application process, there are usually various issues that need to be optimized, such as system stability, system data security, and system operational flexibility.

Due to the importance of data to the middle platform, incorrect configuration of data will lead to frequent production accidents, which will affect customer experience and cause corporate revenue losses. Therefore, ensuring the correctness, persistence and integrity of data will be an important issue that needs to be solved urgently. For the middle platform data, the data volume is large and the data types are diverse. At the same time, it is necessary to ensure the consistency and integrity of data in multiple Internet (Inetnret) hypertext distributed information system (World Wide Web, Web) applications. Most of the existing data processing solutions are only applicable to version control when the data is of a single type and the data volume is small, or the data operation log is synchronized to a distributed open source database (Hbase) for storage, but it is only a backup of the data and cannot be used in real time. Therefore, the existing data version control method cannot be applied to the middle platform data.

Summary of the invention

The present application provides a data processing method, device, electronic device and storage medium that can ensure the consistency, integrity, correctness and persistence of data in multiple distributed information system applications, and make the data effective in each service in real time.

On the one hand, the present application provides a data processing method, which is applied to a middle-office system, wherein the middle-office system is communicatively connected to a peripheral system, and the method includes: sending data update notification information to the peripheral system, wherein the data update notification information is used to notify the peripheral system to receive the pending data in the pending data set from a preset object storage medium; if it is determined that the peripheral system has received all the pending data in the pending data set, sending the data validity notification information to the peripheral system, wherein the data validity notification information is used to notify the peripheral system to validate all the pending data that have been received; if it is determined that the peripheral system has validated all the pending data in the pending data set, updating the data processing task status corresponding to the pending data set to validated.

In a possible implementation of the present application, before sending the data update notification information to the peripheral system, the method also includes: configuring multiple data to be processed to obtain the data set to be processed; packaging the data set to be processed, and uploading the configured data set to be processed to the object storage medium.

In a possible implementation of the present application, the sending of the data update notification information to the peripheral system includes: sending the data update notification information to a data update message queue, so as to send the data update notification information to the peripheral system through the data update message queue.

In a possible implementation of the present application, if it is determined that the peripheral system has received all the pending data in the pending data set, before sending the data effectiveness notification information to the peripheral system, the method also includes: receiving data received information from the callback status message queue, wherein, when the peripheral system completes the task of receiving one of the pending data in the pending data set, the callback status message queue receives the data received information from the peripheral system once; if all the data received information in the callback status message queue has been received, it is determined that the peripheral system has received all the pending data in the pending data set.

In a possible implementation of the present application, if it is determined that the peripheral system has received all the data to be processed in the data set to be processed, sending the data validity notification information to the peripheral system includes: if it is determined that the peripheral system has received all the data to be processed in the data set to be processed, sending the data validity notification information to the data validity message queue, so as to send the data validity notification information to the peripheral system through the data validity message queue.

In a possible implementation of the present application, if it is determined that the peripheral system has validated all the pending data in the pending data set, before updating the data processing task status corresponding to the pending data set to be validated, the method also includes: receiving data validation information from the callback status message queue, wherein when the peripheral system completes the validation task of one of the pending data in the pending data set, the callback status message queue receives the data validation information from the peripheral system; if the data validation information in the callback status message queue is received, it is determined that the peripheral system has validated all the pending data in the pending data set.

On the other hand, the present application provides a data processing method, which is applied to a peripheral system, and the peripheral system is communicatively connected to the middle-office system. The method includes: receiving data update notification information; receiving the to-be-processed data in the to-be-processed data set from a preset object storage medium according to the data update notification information; when the to-be-processed data is received, sending data received information to the middle-office system; receiving data validity notification information; according to the data validity notification information, validating all the to-be-processed data that have been received; if all the to-be-processed data in the to-be-processed data set have been valid, updating the data version management table in the preset database.

In a possible implementation of the present application, the data version management table includes a version identification mapping table, the data to be processed includes a business key, and the data set to be processed includes a current data version identification. After receiving the data to be processed in the data set to be processed from a preset object storage medium according to the data update notification information, the method also includes: synchronously saving the business key and the current data version identification to the version identification mapping table.

In a possible implementation of the present application, the data version management table also includes a valid version identification linked list and a business key mapping table. If all the data to be processed in the data set to be processed have become effective, the data version management table in the preset database is updated, including: adding the current data version identification corresponding to the effective data set to be processed to the valid version identification linked list; generating the business key mapping table based on the valid version identification linked list and the version identification mapping table.

In a possible implementation of the present application, the business key mapping table includes a basic data version identifier, and the method further includes: querying from the valid version identifier linked list to obtain first target data within a first target historical time, where the first target historical time is N days before the last data update, where N is a natural number; matching from the version identifier mapping table to obtain a target business key associated with the first target data and the latest version identifier data of the target business key; and matching the target business key associated with the first target data to the latest version identifier data of the target business key; The latest version identification data of the business key is used as the new basic data version identification of the business key mapping table.

In a possible implementation of the present application, the method also includes: querying from the version identification mapping table to obtain second target data within a second target historical time, where the second target historical time is M days before the last data update, and M is a natural number; matching from the valid version identification linked list to obtain at least one target version identification data associated with the second target data; filtering at least one target version identification data that is in an effective state to obtain at least one filtered target version identification data; deleting all data in the version identification mapping table that is associated with at least one filtered target version identification data.

In a possible implementation of the present application, the method also includes: obtaining a historical data version switching instruction; deleting the specified version identification data in the valid version identification linked list according to the historical data version switching instruction, and obtaining the valid version identification linked list after deleting the specified version identification data; and regenerating the business key mapping table according to the valid version identification linked list after deleting the specified version identification data and the version identification mapping table.

On the other hand, the present application provides a data processing device, which is applied to a middle-office system, and the middle-office system is communicatively connected to a peripheral system. The data processing device includes: an update information sending module, which is used to send data update notification information to the peripheral system, and the data update notification information is used to notify the peripheral system to receive the pending data in the pending data set from a preset object storage medium; an effectiveness information sending module, which is used to send the data effectiveness notification information to the peripheral system if it is determined that the peripheral system has received all the pending data in the pending data set, and the data effectiveness notification information is used to notify the peripheral system to validate all the pending data that have been received; a task status updating module, which is used to update the data processing task status corresponding to the pending data set to effective if it is determined that the peripheral system has validated all the pending data in the pending data set.

On the other hand, the present application provides a data processing device, which is applied to a peripheral system, and the peripheral system is communicatively connected to the middle-office system. The data processing device includes: an update information receiving module, which is used to receive data update notification information; a data receiving module, which is used to receive the pending data in the pending data set from a preset object storage medium according to the data update notification information; a received information sending module, which is used to send data received information to the middle-office system when the pending data is received; an effective information receiving module, which is used to receive data effective notification information; a data effective module, which is used to effective all the pending data that have been received according to the data effective notification information; a data table updating module, which is used to update the data version management table in the preset database if all the pending data in the pending data set have been effective.

On the other hand, the present application provides an electronic device, comprising: one or more processors; a memory; and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the data processing method described in any of the above embodiments.

On the other hand, the present application provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the data processing method described in any of the above embodiments.

The present application sends data update notification information to the peripheral system, and the data update notification information is used to notify the peripheral system to receive the pending data in the pending data set from the preset object storage medium; if it is determined that the peripheral system has received all the pending data in the pending data set, a data validity notification information is sent to the peripheral system, and the data validity notification information is used to notify the peripheral system to validate all the pending data received; if it is determined that the peripheral system has validated all the pending data in the pending data set Data, updates the data processing task status corresponding to the set of data to be processed to effective, that is, this application stores the data to be processed in an object storage medium that is available for external download, which is easy to expand and compatible, and convenient for processing large amounts of data. It determines that the task is allowed to take effect only after the peripheral system has received all the data to be processed, ensuring the atomic effectiveness of large quantities of different types of data, and ensuring the correctness, persistence and integrity of the data; it implements data version management, retains historical data, effectively reduces the risks brought about by the configuration process, and ensures the persistence and integrity of the data of the middle-office system and the peripheral system.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

FIG1 is a schematic diagram of an application scenario of a middle platform system provided in an embodiment of the present application.

FIG. 2 is a schematic diagram of an application scenario of a data processing system provided in an embodiment of the present application.

FIG3 is a flow chart of a data processing method provided in an embodiment of the present application.

FIG. 4 is a flow chart of a data processing method provided in another embodiment of the present application.

FIG5 is a flow chart of a data processing method provided in another embodiment of the present application.

FIG6 is a schematic diagram of the structure of a data processing device provided in another embodiment of the present application.

FIG. 7 is a schematic diagram of the structure of a data processing device provided in another embodiment of the present application.

FIG. 8 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

Mode for Carrying Out the Invention

The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

In the description of the present application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

In this application, the word "exemplary" is used to mean "serving as an example, illustration, or description." Any embodiment described in this application as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. The following description is given to enable any person skilled in the art to implement and use the present application. In the following description, details are listed for the purpose of explanation. It should be understood that one of ordinary skill in the art can recognize that the present application can be implemented without these specific details. In other instances, well-known structures and processes are not described in detail to avoid obscuring the present application with unnecessary detail. Therefore, the present application is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein.

The middle platform system, literally speaking, is located between the front-end system and the back-end system. As shown in Figure 1, the front-end system can be multiple peripheral systems. During the application process, data flows from the back-end system into the middle platform system to complete the storage, calculation, and product packaging of massive data, forming the core data capabilities of the enterprise, and providing strong support for the customized innovation of multiple peripheral systems based on data and the continuous evolution of the business middle platform based on data feedback. For the data of the middle platform system, the data volume is large and the data types are diverse. At the same time, the consistency and integrity of the data in multiple distributed network applications must be guaranteed. Therefore, the existing data processing methods are not applicable to the data of the middle platform system at all.

In order to solve the above problems, the embodiments of the present application provide a data processing method, device, electronic device and storage medium, which are described in detail below.

The executor of the data processing method of the embodiment of the present application may be the data processing device provided in the embodiment of the present application, or different types of electronic devices such as a server device, a physical host or a user equipment (UE) that integrates the data processing device, wherein the data processing device may be implemented in hardware or software, and the UE may specifically be a terminal device such as a smart phone, a tablet computer, a laptop computer, a PDA, a desktop computer or a personal digital assistant (PDA).

The electronic device may work in a single operation mode or in a device cluster mode.

As shown in Figure 2, Figure 2 is a scene diagram of a data processing system provided by an embodiment of the present application. Among them, the data processing system may include an electronic device 100 for completing the data processing method, and a data processing device is integrated in the electronic device 100. For example, the electronic device can be used to send data update notification information to the peripheral system, and the data update notification information is used to notify the peripheral system to receive the pending data in the pending data set from the preset object storage medium; if it is determined that the peripheral system has received all the pending data in the pending data set, send data validity notification information to the peripheral system, and the data validity notification information is used to notify the peripheral system to validate all the received pending data; if it is determined that the peripheral system has validated all the pending data in the pending data set, update the data processing task status corresponding to the pending data set to be validated.

In addition, as shown in FIG. 2 , the data processing system may further include a memory 200 for storing data, such as business data related to the business middle station and other data.

It should be noted that the scenario diagram of the data processing system shown in Figure 2 is merely an example. The data processing system and scenario described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of data processing systems and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.

Next, we will introduce the data processing method provided by the embodiment of the present application. In the embodiment of the present application, an electronic device is used as the execution subject. For the sake of simplicity and ease of description, the execution subject will be omitted in the subsequent method embodiments. The data processing method includes: sending data update notification information to the peripheral system, the data update notification information is used to notify the peripheral system to receive the pending data in the pending data set from the preset object storage medium; if it is determined that the peripheral system has received all the pending data in the pending data set, sending data validity notification information to the peripheral system, the data validity notification information is used to notify the peripheral system to validate all the received pending data; if it is determined that the peripheral system has validated all the pending data in the pending data set, it will be processed together with the pending data. The data processing task status corresponding to the data collection is updated to effective.

This application stores the data to be processed in an object storage medium that is available for external download. It is easy to expand and compatible, and is convenient for processing large amounts of data. It allows the task to take effect only after it is determined that the peripheral system has received all the data to be processed, thereby ensuring the atomic effectiveness of large quantities of different types of data, and ensuring the correctness, persistence and integrity of the data; it implements data version management, retains historical data, effectively reduces the risks brought about by the configuration process, and ensures the persistence and integrity of the data of the middle-end system and the peripheral system.

As shown in Figures 3 and 4, Figures 3 and 4 are both flowcharts of the data processing method provided in an embodiment of the present application. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in an order different from that here. The data processing method is applied to the middle station system, which is communicatively connected with the peripheral system, as shown in Figure 3, and the method specifically includes steps S201 to S203.

S201. Send data update notification information to a peripheral system, where the data update notification information is used to notify the peripheral system to receive the to-be-processed data in the to-be-processed data set from a preset object storage medium.

The data to be processed is the data that needs to be configured and effective in the peripheral system; the data set to be processed is a collection of multiple data to be processed in the same batch.

In this embodiment, the multiple data to be processed in the data set to be processed may be data of the same type or data of different types. The data to be processed in the data set to be processed may include various business data. For example, the data to be processed may be business data applied to prices, or business data applied to products and timeliness, etc. This embodiment does not make any specific limitations on this.

In order to realize the storage and configuration of a large number of different types of data to be processed, and to facilitate the synchronization and real-time effectiveness of different types of data in multiple different peripheral systems at the same time, in this embodiment, the middle-office system generates a file for the configuration of the data set to be processed and uploads it to the object storage medium. When the middle-office system notifies multiple peripheral systems that there is a data update, the corresponding files are sent to the multiple peripheral systems through the object storage medium.

Therefore, in this embodiment, before sending data update notification information to the peripheral system, the method also includes: configuring multiple data to be processed to obtain a set of data to be processed; packaging the set of data to be processed, and uploading the configured set of data to be processed to the object storage medium.

Configure multiple data to be processed to obtain a set of data to be processed, specifically:

Users can create data configuration tasks in the middle-office system and configure different business keys according to different data to be processed. The business key is a user-defined data feature of the data to be processed. In this embodiment, the same business key can also be configured for the same type of data to be processed. For example, the same business key is configured for overtime data and service time data. Therefore, one business key may contain multiple data to be processed. The business key of the data to be processed can be configured according to specific needs, and this embodiment does not make any specific limitations on this.

Other information is configured for the multiple data to be processed as needed. For example, other information such as data type information and current configuration time information can be configured for the multiple data to be processed as needed. Finally, a data set to be processed containing multiple data to be processed is obtained, and a current data version identifier corresponding to the data set to be processed is generated. The current data version identifier is the version number (batch number) of the data set to be processed.

Package the data set to be processed and upload the configured data set to be processed to the object storage medium. Specifically:

The data set to be processed corresponds to multiple version files. The format of the version file can be a compressed package. The data set to be processed is packaged, and the configured data set to be processed is uploaded to the object storage medium.

That is, in a newly created data configuration task, the data to be processed can be regarded as a business data table, and a business data table is regarded as a version file. That is, each time the peripheral system updates the business data, a version file corresponding to the business data table is generated. Multiple version files are merged and compressed, and a compressed package corresponds to a current data version identifier. Each piece of data to be processed is written in protostuff format. In this embodiment, other configuration methods can also be used, and this embodiment does not specifically limit this. When the object storage medium sends the data to be processed to the peripheral system, each sending task corresponds to a file. In this embodiment, the data to be processed is distributed to the peripheral system through the data distribution entity.

In this embodiment, the object storage medium may be a database using object storage service technology (Object Storage Service, OSS), wherein OSS is a distributed object storage service that provides an object storage service in the form of a key-value pair. A user may uniquely obtain the content of an object according to the name (Key) of the object. In this embodiment of the application, the data to be processed may be stored in the form of an object in OSS, and a business key, a current data version identifier, and a data task status may be set for the data set to be processed. The data task status is initially in an ineffective state, and the current data version identifier of the data to be processed is the name (Key) of the data set to be processed.

In this embodiment, in order to simultaneously ensure the consistency and integrity of data in multiple distributed web applications, as well as the synchronization and atomicity of different types of data, that is, to ensure that the receiving operations of different types of data between the middle-end system and multiple different peripheral systems are all successfully executed before the next operation can be performed, if the receiving operation of any of the data is unsuccessful, any operation that has been executed must be revoked, and the data task status must be returned to the initial state. The middle-end system and the peripheral system in this application use a message queue to communicate messages. Therefore, in this embodiment, sending data update notification information to the peripheral system specifically includes: sending data update notification information to the data update message queue, so as to send the data update notification information to the peripheral system through the data update message queue.

That is, in this embodiment, the middle-office system acts as the producer of the data update message queue, and the peripheral system acts as the consumer of the data update message queue. The data update notification information is sent to the data update message queue through the middle-office system, and the peripheral system receives the data update notification information from the data update message queue, so as to realize the middle-office system sending data update notification information to the peripheral system.

Data update notification information can contain simple text strings or complex embedded objects.

Specifically, the fields of data update notification information can be configured as follows:

Task type: product configuration, whitelist configuration, price configuration, timeliness configuration, etc.

The current data version identifier of the data set to be processed;

Message-Digest Algorithm (MD5) of the data set to be processed: used for encryption and decryption;

Download address of the data set to be processed;

Data status to be processed: sent only, effective immediately;

Contents of the data set to be processed: descriptions related to the contents of the data to be processed;

The data structure of the data set to be processed can be configured as:

File name: table name;

Class name: the corresponding entity name;

The total number of data rows in the data set to be processed: the number of data rows to be processed that are transferred.

The data update notification information may also be configured with other fields, which are not specifically limited in this embodiment.

The data update message queue is a container for storing data update notification messages during the transmission process of data update notification information. Its main purpose is to provide routing and ensure the delivery of messages. The data update message queue can use any one of Kafka, RabbitMQ or ActiveMQ, and this embodiment does not make specific limitations on this.

S202: If it is determined that the peripheral system has received all the to-be-processed data in the to-be-processed data set, data validation notification information is sent to the peripheral system, where the data validation notification information is used to notify the peripheral system to validate all the received to-be-processed data.

After the peripheral system completes the data receiving task of the data to be processed, it will send data reception information to the middle-office system to notify the middle-office system that the peripheral system has completed data reception.

Therefore, if it is determined that the peripheral system has received all the pending data in the pending data set, before sending data effectiveness notification information to the peripheral system, the method also includes: receiving data received information from the callback status message queue, wherein, when the peripheral system completes the task of receiving one pending data in the pending data set, the callback status message queue receives data received information from the peripheral system once; if all the data received information in the callback status message queue has been received, it is determined that the peripheral system has received all the pending data in the pending data set.

In this embodiment, the peripheral system acts as the producer of the callback status message queue, and the middle-stage system acts as the consumer of the callback status message queue. After the peripheral system completes the data reception task of one of the pending data sets, it will send a data reception message to the callback status message queue. The middle-stage system acts as the subscriber of the callback status message queue. The peripheral system sends a data reception message to the callback status message queue, and the middle-stage system receives the data reception message sent by the callback status message queue to determine that the peripheral system has completed data reception. The data reception message can contain simple text strings or complex embedded objects.

Specifically, the fields of the data received information can be configured as follows:

Unique identifier of the data set to be processed;

MD5 of the data to be processed;

Data status to be processed: data received, data effective, data processing exception, etc.

The data received information may also be configured with other fields, which is not specifically limited in this embodiment.

In this embodiment, the callback status message queue may adopt any one of Kafka, RabbitMQ or ActiveMQ, and this embodiment does not make any specific limitation to this.

In order to ensure data consistency and accuracy between the middle-office system and multiple peripheral systems, in this embodiment, the middle-office system allows each peripheral system to validate the data to be processed only after it determines that the peripheral system has completed all data receiving tasks. That is, if it is determined that the peripheral system has received all the data to be processed in the data set to be processed, the data validation notification information is sent to the peripheral system.

In this embodiment, if it is determined that the peripheral system has received all the data to be processed in the data set to be processed, sending data validity notification information to the peripheral system specifically includes:

If it is determined that the peripheral system has received all the to-be-processed data in the to-be-processed data set, data validity notification information is sent to the data validity message queue, so as to send the data validity notification information to the peripheral system through the data validity message queue.

That is, in this embodiment, the middle-office system acts as the producer of the data validity message queue, and the peripheral system acts as the consumer of the data validity message queue. The data validity notification information is sent to the data validity message queue through the middle-office system, and the peripheral system receives the data validity notification information from the data validity message queue, so as to realize that the middle-office system sends the data validity notification information to the peripheral system.

Data validation notification information can contain simple text strings or complex embedded objects.

Specifically, the fields of data effectiveness notification information can be configured as follows:

The unique identifier of the data set to be processed;

Download address of the data set to be processed;

Data status to be processed: sent only, effective immediately;

The data structure of the data set to be processed can be configured as:

File name: table name;

Class name: the corresponding entity name;

The data validity notification information may also be configured with other fields, which are not specifically limited in this embodiment.

The data validation message queue may adopt any one of Kafka, RabbitMQ or ActiveMQ, and this embodiment does not specifically limit this.

S203: If it is determined that the peripheral system has validated all the data to be processed in the data set to be processed, the data processing task status corresponding to the data set to be processed is updated to validated.

After the peripheral system completes the data validation task for the data to be processed, it will send data validation information to the middle-office system to notify the middle-office system that the peripheral system has completed the data validation.

Therefore, in this embodiment, if it is determined that the peripheral system has validated all the pending data in the pending data set, before updating the data processing task status corresponding to the pending data set to validated, the method also includes: receiving data validation information from the callback status message queue, wherein when the peripheral system completes the validation task of one pending data in the pending data set, the callback status message queue receives data validation information from the peripheral system once; if all the data validation information in the callback status message queue has been received, it is determined that the peripheral system has validated all the pending data in the pending data set.

The middle-office system is a subscriber to the callback status message queue. When the peripheral system sends a data-validated message to the callback status message queue once, the middle-office system will receive the data-validated message sent by the callback status message queue. If all the data-validated messages in the callback status message queue have been received, it is determined that the peripheral system has validated all the pending data in the pending data set.

The callback status message queue proposed in this step and the callback status message queue proposed in step S202 can be the same message queue, the difference between them is that the pending data status in the data received information and the data effective information is different. When the status field of the pending data in the callback status message queue shows that the data has been received, the middle-office system decouples the messages in the callback status message queue and determines that the peripheral system has received the pending data. When the status field of the pending data in the callback status message queue shows that the data has been effective, the middle-office system decouples the messages in the callback status message queue and determines that the peripheral system has effective the pending data.

In this embodiment, the callback status message queue may also be a different message queue, and this application does not make any specific limitation on this.

As shown in Figures 4 and 5, Figures 4 and 5 are both flowchart diagrams of the data processing method provided in an embodiment of the present application. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in an order different from that here. The data processing method is applied to a peripheral system, and the peripheral system is communicatively connected with the middle station system. As shown in Figure 5, the method specifically includes steps S301 to S306.

S301. Receive data update notification information.

S302: Receive the to-be-processed data in the to-be-processed data set from a preset object storage medium according to the data update notification information.

S303. When the data to be processed is received, the data reception information is sent to the middle station system.

S304: Receive data validation notification information.

S305. Validate all received data to be processed according to the data validation notification information.

In this embodiment, since steps S201 to S203 have disclosed the receiving and sending of data update notification information through the data update message queue, the receiving and sending of data validity notification information through the data validity message queue, and the receiving and sending of data received information and data valid information through the callback status message queue between the middle-office system and the peripheral system, steps S301 to S305 are actually the steps corresponding to steps S201 to S203, and therefore, they are not repeated in this embodiment.

S306: If all the data to be processed in the data set to be processed have become effective, the data version management table in the preset database is updated.

After the peripheral system has validated all the received data to be processed according to the data validation notification information, in order to retain historical data, ensure the persistence and integrity of system data, and realize data version control and real-time validation of data, the data version management table in the preset database can be updated when all the data to be processed in the data set to be processed have taken effect.

In this embodiment, the data version management table includes a valid version identification chain table. To improve data query and validation performance, the peripheral system can also maintain two additional tables, that is, the data version management table can also include a version identification mapping table and a business key mapping table.

(1) Valid version identification list:

Used to record the basic data version identifier and the current data version identifier. The basic data version identifier is the version identifier of the initial data initially configured by the peripheral system;

Write data when the pending data set takes effect;

The fields include the current data version identifier, data effective time, and data status;

The database is stored in the peripheral system end, wherein the database of the peripheral system end may be a database using object storage service technology, or may be other free and open source databases, which is not specifically limited in this embodiment.

(2) Version identification mapping table:

It is used to record the business key set corresponding to the current data version identifier, but does not record the basic data version identifier;

Write data when the data set to be processed is received;

The fields include the corresponding current data version identifier, business key type, business key, and data receiving time;

(3) Business key mapping table:

It is derived by combining the valid version identifier chain list and the version identifier mapping table;

Used to record the basic data version identifier and the data version identifier currently in effect for the business key;

Write data when the pending data set takes effect;

The fields include the business key, the current data version identifier corresponding to the business key, and the data effective time, etc.

The current data version of the data to be processed is identified as the first prefix of the remote cache key in the database, remote cache and local cache stored on the peripheral system side; wherein, the database on the peripheral system side may be a database using object storage service technology, or other free and open source databases, which is not specifically limited in this embodiment.

The local cache is the local cache of the peripheral system.

Remote cache can use remote dictionary server (Remote Dictionary Server, Redis). Redis has a high-performance Key-Value database storage mechanism, and supports data persistence, no data loss after restart, supports distributed deployment, supports data backup and other features. Remote cache can also use any remote cache component, which is not specifically limited in this embodiment.

In this embodiment, the table structure of the local cache and the remote cache can be set as follows:

p2cache_config (cache configuration table): stores the refresh rules of pending data, such as key field, value field, refresh frequency, and filter conditions;

p2cache_key: stores the key value of the data set to be processed;

p2cache_detail: stores the value of the data set to be processed.

During the application process, when the peripheral system needs to use the current business key data, it will first find its latest current data version identifier, and then obtain the latest configuration data through the current data version identifier and business key, so as to realize data version control and real-time data effectiveness, and then realize the correctness, persistence and integrity of the data, and make the data effective in real time in each service.

The following is a detailed introduction to the update method of "updating the data version management table in the preset database".

In another embodiment of the present application, after receiving the to-be-processed data in the to-be-processed data set from a preset object storage medium according to the data update notification information, the method further includes: synchronously saving the business key and the current data version identifier to a version identifier mapping table.

Specifically, after the peripheral system successfully receives the data to be processed, it parses the mapping relationship between the business key of the data to be processed and the current data version identifier, and saves the mapping relationship between the business key and the current data version identifier to the version identifier mapping table; at the same time, the local cache and remote cache of the peripheral system are refreshed, and the business key of the data set to be processed and the current data version identifier of the data to be processed are synchronously saved to the remote cache.

In another embodiment of the present application, if all the data to be processed in the data set to be processed have been effective, the data version management table in the preset database is updated, including: adding the current data version identifier corresponding to the effective data set to be processed to the valid version identifier linked list; generating a business key mapping according to the valid version identifier linked list and the version identifier mapping table; surface.

The current data version identifier corresponding to the valid set of data to be processed is added to the valid version identifier linked list. Specifically, after the peripheral system successfully validates the data to be processed, the current data version identifier of the data to be processed is added to the end of the valid version identifier linked list.

A business key mapping table is generated according to the valid version identifier chain list and the version identifier mapping table, specifically: the valid version identifier chain list is traversed in descending order to obtain the current data version identifier that is currently in effect; the version identifier mapping table is matched according to the current data version identifier; the version identifier mapping table is traversed to determine the business key; when the business key does not exist in the preset business key mapping table, the business key is written into the business key mapping table; when the business key does not exist in the preset business key mapping table, no operation is performed, and finally a business key mapping table is generated; at the same time, the local cache and remote cache of the peripheral system are refreshed, and the business key of the data set to be processed and the current data version identifier of the data to be processed are synchronously saved to the remote cache.

In another embodiment of the present application, refreshing the local cache and the remote cache of the peripheral system specifically includes:

In this embodiment, in order to simplify the description, the local cache and the remote cache are both referred to as caches below;

1. Producer scheduled task: The producer queries the cache p2cache_config configuration every 10 seconds, queries the corresponding business table according to the configuration, and inserts the pending data that needs to be refreshed into the p2cache_key table and p2cache_detail table; 2. Consumer scheduled task: The consumer queries the p2cahce_key table and p2cache_detail data every 10 seconds and refreshes it to the cache.

In another embodiment of the present application, in order to improve service performance, the business key mapping table is updated to the local cache in real time.

Update the business key mapping table to the local cache in real time, including:

First full startup: query the database to obtain the business key and the corresponding current data version identifier, and store the queried business key and the corresponding current data version identifier in the local cache business key mapping table;

Subsequent increments: When data distribution and cache refresh are completed, notify the local update of the mapping relationship between the business key and the current data version identifier.

In the present application, in order to avoid too many current data version identifiers being saved, that is, to avoid too much dirty data in the cache causing data confusion, data anomalies and affecting system performance, the peripheral system cleans up dirty data through regular pruning. Regular pruning may include: merging historical data version identifiers into basic data version identifiers; regularly cleaning up expired data version identifier caches. Among them, the historical data version identifier is a new data version identifier recorded on the basic data version identifier each time the peripheral system updates business data, and the recorded new data version identifier includes the current data version identifier. In this embodiment, other data pruning methods may also be included, and this application does not specifically limit this.

Therefore, in another embodiment of the present application, the data processing method further includes: merging the historical data version identifier into the basic data version identifier.

Merge the historical data version identifier into the basic data version identifier, including:

Query and obtain the first target data within the first target historical time from the valid version identification linked list;

The first target historical time is N days before the last data update, where N is a natural number and the value of N is set as required, such as the previous 7 days, which is not specifically limited in this embodiment;

The first target data includes all data in the valid version identifier chain list within the first target historical time. For example, the first target data may include the data version identifier within the first target historical time, the data effective time of the data version identifier, time and data status, etc. Obtain the target business key associated with the first target data and the latest version identification data of the target business key by matching from the version identification mapping table;

The latest version identification data of the target business key is used as the new basic data version identification of the business key mapping table, and the business data table and cache are refreshed.

In this embodiment, it should be noted that if a new basic data version identifier already exists in the business key mapping table, the existing new basic data version identifier is deleted, and the business data table and cache are refreshed.

In another embodiment of the present application, the method further includes: regularly clearing the expired data version identification cache.

Regularly clean up the expired data version identifier cache, specifically:

Obtain the second target data within the second target historical time from the version identification mapping table;

The second target historical time is M days before the last data update, where M is a natural number and the value of M is set as required, such as the previous 7 days, which is not specifically limited in this embodiment;

The second target data includes all data in the version identification mapping table within the second target historical time. Exemplarily, the second target data may include data version identification, business key type, business key, and data receiving time within the second target historical time. At least one target version identification data associated with the second target data is obtained by matching from the valid version identification linked list;

Filtering the target version identification data in the effective state in the at least one target version identification data to obtain the at least one target version identification data after filtering;

Delete all data in the version identification mapping table that is associated with at least one filtered target version identification data, and refresh the business data table and cache.

In the present application, the execution interval of "merging historical data version identifiers into basic data version identifiers" and "regularly clearing expired data version identifier cache" can be set according to actual conditions, and this embodiment does not make specific limitations.

In another embodiment of the present application, since the peripheral system may need to roll back to a historical data version after updating the data, in order to facilitate the peripheral system to switch the data version, the data processing method further includes:

Obtaining a historical data version switching instruction, where the historical data version switching instruction may be sent by a user to a device executing the data processing method, or may be automatically generated periodically by the device executing the data processing method, which is not specifically limited in this embodiment;

According to the historical data version switching instruction, the specified version identification data in the valid version identification chain list is deleted to obtain the valid version identification chain list after deleting the specified version identification data;

The business key mapping table is regenerated according to the valid version identification chain list and the version identification mapping table after the specified version identification data is deleted.

In this embodiment, the historical data version switching instruction includes a "rollback specified version identification data instruction" and a "rollback to specified version identification data instruction", wherein, according to the historical data version switching instruction, the specified version identification data in the valid version identification linked list is deleted, and the valid version identification linked list after deleting the specified version identification data is obtained, specifically:

If the historical data version switching instruction is a "roll back specified version identification data instruction", the specified version identification data required to be deleted by the "roll back specified version identification data instruction" in the valid version identification chain table is deleted, and the valid version identification chain table after deleting the specified version identification data is obtained, and the valid version identification chain table after deleting the specified version identification data and the version identification mapping are used to convert the specified version identification data into a valid version identification chain table. Shooting table, regenerate business key mapping table, and refresh cache;

If the historical data version switching instruction is the "roll back to the specified version identification data instruction", then all version identification data that are arranged after the specified version identification data and that the "roll back to the specified version identification data instruction" requires to be deleted in the valid version identification linked list will be deleted, and based on the valid version identification linked list and the version identification mapping table that have deleted all version identification data that are arranged after the specified version identification data, the business key mapping table will be regenerated and the cache will be refreshed.

The above method further realizes the control of data version and the real-time effectiveness of data, retains historical data, effectively reduces the risks brought about by the configuration process, and ensures the persistence and integrity of system data.

In another embodiment of the present application, in order to facilitate the peripheral system to query data, the data processing method further includes:

According to the business key generation strategy, the business key value is obtained;

The data version identifier is matched in the version identifier mapping table according to the business key. When there is no match result, the data version identifier is the basic data version identifier. The cache is queried by the data version identifier and the business key.

In order to better implement the data processing method in the embodiment of the present application, on the basis of the data processing method, a data processing device 400 is also provided in the embodiment of the present application, which is applied to the middle-office system, and the middle-office system is communicatively connected with the peripheral system. As shown in Figure 6, the data processing device 400 includes an update information sending module 401, an effectiveness information sending module 402 and a task status update module 403.

The update information sending module 401 is used to send data update notification information to the peripheral system, and the data update notification information is used to notify the peripheral system to receive the to-be-processed data in the to-be-processed data set from a preset object storage medium.

The validity information sending module 402 is used to send data validity notification information to the peripheral system if it is determined that the peripheral system has received all the to-be-processed data in the to-be-processed data set. The data validity notification information is used to notify the peripheral system to validate all the received to-be-processed data.

The task status updating module 403 is used to update the data processing task status corresponding to the to-be-processed data set to be effective if it is determined that the peripheral system has effective all the to-be-processed data in the to-be-processed data set.

In some embodiments of the present application, the data processing device 400 also includes a data configuration module 404, which is specifically used to: configure multiple data to be processed to obtain a data set to be processed; package the data set to be processed, and upload the configured data set to be processed to the object storage medium.

In some embodiments of the present application, the update information sending module 401 is specifically used to: send data update notification information to the data update message queue, so as to send the data update notification information to the peripheral system through the data update message queue.

In some embodiments of the present application, the data processing device 400 also includes a data reception determination module 405, which is specifically used to: receive data received information from the callback status message queue, wherein when the peripheral system completes the reception task of one of the to-be-processed data in the to-be-processed data set, the callback status message queue receives data received information from the peripheral system once; if all the data received information in the callback status message queue is received, it is determined that the peripheral system has received all the data in the to-be-processed data set. Data to be processed.

In some embodiments of the present application, the effectiveness information sending module 402 is specifically used to: if it is determined that the peripheral system has received all the data to be processed in the data set to be processed, send data effectiveness notification information to the data effectiveness message queue, so as to send the data effectiveness notification information to the peripheral system through the data effectiveness message queue.

In some embodiments of the present application, the data processing device 400 also includes a data effectiveness determination module 406, which is specifically used to: receive data effectiveness information from the callback status message queue, wherein when the peripheral system completes the effectiveness task of one of the pending data in the pending data set, the callback status message queue receives the data effectiveness information from the peripheral system; if the data effectiveness information in the callback status message queue has been received, it is determined that the peripheral system has effective all the pending data in the pending data set.

In order to better implement the data processing method in the embodiment of the present application, on the basis of the data processing method, a data processing device 500 is also provided in the embodiment of the present application, which is applied to the peripheral system, and the peripheral system is communicatively connected with the middle system. As shown in Figure 7, the data processing device 500 includes an update information receiving module 501, a data receiving module 502, a received information sending module 503, an effective information receiving module 504, a data effective module 505 and a data table update module 506.

The update information receiving module 501 is used to receive data update notification information.

The data receiving module 502 is used to receive the to-be-processed data in the to-be-processed data set from a preset object storage medium according to the data update notification information.

The receiving information sending module 503 is used to send data reception information to the middle station system when the data to be processed is received.

The validity information receiving module 504 is used to receive data validity notification information.

The data validation module 505 is used to validate all received data to be processed according to the data validation notification information.

The data table updating module 506 is used to update the data version management table in the preset database if all the data to be processed in the data set to be processed have become effective.

In some embodiments of the present application, the data version management table includes a version identification mapping table, the data to be processed includes a business key, the data set to be processed includes a current data version identification, and the data processing device 500 also includes a synchronous update module 507, which is specifically used to: synchronously save the business key and the current data version identification to the version identification mapping table.

In some embodiments of the present application, the data version management table also includes a valid version identifier linked list and a business key mapping table, and the data table update module 506 is specifically used to: add the current data version identifier corresponding to the effective set of data to be processed to the valid version identifier linked list; generate a business key mapping table based on the valid version identifier linked list and the version identifier mapping table.

In some embodiments of the present application, the business key mapping table includes a basic data version identifier, and the data table update module 506 is also used to: query from the valid version identifier linked list to obtain the first target data within the first target historical time, the first target historical time is N days before the last data update, and N is a natural number; match the target business key associated with the first target data and the latest version identification data of the target business key from the version identification mapping table; and use the latest version identification data of the target business key as the new basic data version identifier of the business key mapping table.

In some embodiments of the present application, the data table update module 506 is further used to: query from the version identification mapping table to obtain the second target data within the second target historical time, where the second target historical time is M days before the last data update, where M is a natural number; match from the valid version identification linked list to obtain at least one target version identification data associated with the second target data; filter the target version identification data in the effective state from the at least one target version identification data to obtain the filtered at least one target version identification data. deleting all data in the version identification mapping table associated with at least one target version identification data after filtering.

In some embodiments of the present application, the data table update module 506 is also used to: obtain historical data version switching instructions; delete specified version identification data in the valid version identification list according to the historical data version switching instructions, and obtain the valid version identification list after deleting the specified version identification data; regenerate the business key mapping table according to the valid version identification list and the version identification mapping table after deleting the specified version identification data.

In another embodiment of the present application, as shown in FIG. 8 , the present application further provides an electronic device 600, which shows a schematic diagram of the structure of the electronic device involved in the embodiment of the present application, specifically:

The electronic device may include components such as a processor 601 with one or more processing cores, a memory 602 with one or more computer-readable storage media, a power supply 603, and an input unit 604. Those skilled in the art will appreciate that the electronic device structure shown in FIG8 does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:

The processor 601 is the control center of the electronic device, and uses various interfaces and lines to connect the various parts of the entire electronic device. By running or executing the software programs and/or modules stored in the memory 602, and calling the data stored in the memory 602, the processor 601 performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 601 may include one or more processing cores; the processor 601 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. Preferably, the processor 601 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 601.

The memory 602 can be used to store software programs and modules. The processor 601 executes various functional applications and data processing by running the software programs and modules stored in the memory 602. The memory 602 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 602 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 602 may also include a memory controller to provide the processor 601 with access to the memory 602.

The electronic device also includes a power supply 603 for supplying power to each component. Preferably, the power supply 603 can be logically connected to the processor 601 through a power management system, so that the power management system can manage charging, discharging, power consumption and other functions. The power supply 603 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components.

The electronic device may further include an input unit 604, which may be used to receive input digital or character information. and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail here.

Specifically in this embodiment, the processor 601 in the electronic device will load the executable files corresponding to the processes of one or more applications into the memory 602 according to the following instructions, and the processor 601 will run the application stored in the memory 602 to implement various functions, as follows:

Sending data update notification information to the peripheral system, where the data update notification information is used to notify the peripheral system to receive the to-be-processed data in the to-be-processed data set from a preset object storage medium;

If it is determined that the peripheral system has received all the to-be-processed data in the to-be-processed data set, data validity notification information is sent to the peripheral system, where the data validity notification information is used to notify the peripheral system to validate all the to-be-processed data that have been received;

If it is determined that the peripheral system has validated all the to-be-processed data in the to-be-processed data set, the data processing task status corresponding to the to-be-processed data set is updated to validated.

Specifically, in another embodiment of the present application, the processor 601 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 602 according to the following instructions, and the processor 601 will run the application programs stored in the memory 602, thereby realizing various functions, as follows:

Receive data update notification information;

Receiving the data to be processed in the data set to be processed from a preset object storage medium according to the data update notification information;

When the data to be processed is received, the data reception information is sent to the middle office system;

Receive data effectiveness notification information;

According to the data effectiveness notification information, all received data to be processed are effective;

If all the data to be processed in the data set to be processed are effective, the data version management table in the preset database is updated.

A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

In some embodiments of the present application, the present application also provides a computer-readable storage medium, which may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc. A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in the data processing method provided in the embodiments of the present application.

In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the detailed description of other embodiments above, and will not be repeated here.

The above is a detailed introduction to a data processing method, device, electronic device and storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

A data processing method, characterized in that it is applied to a middle-end system, wherein the middle-end system is communicatively connected with a peripheral system, and the method comprises:

Sending data update notification information to the peripheral system, wherein the data update notification information is used to notify the peripheral system to receive the to-be-processed data in the to-be-processed data set from a preset object storage medium;

If it is determined that the peripheral system has received all the to-be-processed data in the to-be-processed data set, the data validity notification information is sent to the peripheral system, where the data validity notification information is used to notify the peripheral system to validate all the to-be-processed data that have been received;

If it is determined that the peripheral system has validated all the data to be processed in the data set to be processed, the data processing task status corresponding to the data set to be processed is updated to validated.
The data processing method according to claim 1, characterized in that before sending the data update notification information to the peripheral system, the method further comprises:

configuring a plurality of the data to be processed to obtain the data set to be processed;

The to-be-processed data set is packaged, and the configured to-be-processed data set is uploaded to the object storage medium.
The data processing method according to claim 1 or 2, characterized in that the sending of data update notification information to the peripheral system comprises:

The data update notification information is sent to a data update message queue, so as to send the data update notification information to the peripheral system through the data update message queue.
The data processing method according to any one of claims 1 to 3, characterized in that before sending the data validity notification information to the peripheral system if it is determined that the peripheral system has received all the data to be processed in the data set to be processed, the method further comprises:

Receiving data received information from the callback status message queue, wherein when the peripheral system completes a receiving task of one of the to-be-processed data in the to-be-processed data set, the callback status message queue receives the data received information from the peripheral system once;

If all the data received information in the callback status message queue has been received, it is determined that the peripheral system has received all the to-be-processed data in the to-be-processed data set.
The data processing method according to any one of claims 1 to 4, characterized in that if it is determined that the peripheral system has received all the data to be processed in the set of data to be processed, sending the data validity notification information to the peripheral system comprises:

If it is determined that the peripheral system has received all the to-be-processed data in the to-be-processed data set, the data validity notification information is sent to the data validity message queue, so as to send the data validity notification information to the peripheral system through the data validity message queue.
The data processing method according to any one of claims 1 to 5, characterized in that before updating the data processing task status corresponding to the set of data to be processed to be effective if it is determined that the peripheral system has validated all the data to be processed in the set of data to be processed, the method further comprises:

receiving data validation information from a callback status message queue, wherein when the peripheral system completes a validation task of one of the to-be-processed data in the to-be-processed data set, the callback status message queue receives the data validation information from the peripheral system;

If the data validation information in the callback status message queue has been received, it is determined that the peripheral system has validated all the to-be-processed data in the to-be-processed data set.
A data processing method, characterized in that it is applied to a peripheral system, the peripheral system is communicatively connected with a middle system, and the method comprises:

Receive data update notification information;

Receiving the to-be-processed data in the to-be-processed data set from a preset object storage medium according to the data update notification information;

When the data to be processed is received, the data reception information is sent to the middle station system;

Receive data effectiveness notification information;

According to the data validation notification information, all the received data to be processed are validated;

If all the data to be processed in the data set to be processed are valid, the data version management table in the preset database is updated.
The data processing method according to claim 7, characterized in that the data version management table includes a version identification mapping table, the data to be processed includes a business key, the data set to be processed includes a current data version identification, and after receiving the data to be processed in the data set to be processed from a preset object storage medium according to the data update notification information, the method further includes:

The service key and the current data version identifier are synchronously saved in the version identifier mapping table.
The data processing method according to claim 8, characterized in that the data version management table further includes a valid version identification chain table and a business key mapping table, and if all the data to be processed in the data set to be processed are effective, updating the data version management table in the preset database comprises:

Adding the current data version identifier corresponding to the valid data set to be processed to the valid version identifier linked list;

The service key mapping table is generated according to the valid version identifier chain list and the version identifier mapping table.
The data processing method according to claim 9, wherein the business key mapping table includes a basic data version identifier, and the method further comprises:

Query and obtain the first target data within the first target historical time from the valid version identifier linked list, where the first target historical time is N days before the last data update, where N is a natural number;

Obtaining the target service key associated with the first target data and the latest version identification data of the target service key by matching from the version identification mapping table;

The latest version identification data of the target business key is used as the new basic data version identification of the business key mapping table.
The data processing method according to claim 9 or 10, further comprising:

Obtain second target data within a second target historical time from the version identifier mapping table, where the second target historical time is M days before the last data update, where M is a natural number;

Matching and obtaining at least one target version identification data associated with the second target data from the valid version identification chain list;

Filtering the target version identification data in the effective state in at least one of the target version identification data to obtain the at least one target version identification data after filtering;

All data in the version identification mapping table that is associated with at least one of the filtered target version identification data is deleted.
The data processing method according to any one of claims 9 to 11, further comprising:

Get historical data version switching instructions;

According to the historical data version switching instruction, the specified version identification data in the valid version identification chain list is deleted to obtain the valid version identification chain list after deleting the specified version identification data;

The service key mapping table is regenerated according to the valid version identification chain list and the version identification mapping table after deleting the specified version identification data.
A data processing device, characterized in that it is applied to a middle-end system, the middle-end system is communicatively connected with a peripheral system, and the data processing device comprises:

An update information sending module, used for sending data update notification information to the peripheral system, wherein the data update notification information is used for notifying the peripheral system to receive the to-be-processed data in the to-be-processed data set from a preset object storage medium;

A validation information sending module, used for sending the data validation notification information to the peripheral system if it is determined that the peripheral system has received all the to-be-processed data in the to-be-processed data set, wherein the data validation notification information is used to notify the peripheral system to validate all the to-be-processed data that have been received;

The task status updating module is used to update the data processing task status corresponding to the to-be-processed data set to be effective if it is determined that the peripheral system has effective all the to-be-processed data in the to-be-processed data set.
A data processing device, characterized in that it is applied to a peripheral system, the peripheral system is communicatively connected with a middle-end system, and the data processing device comprises:

An update information receiving module, used for receiving data update notification information;

A data receiving module, configured to receive the to-be-processed data in the to-be-processed data set from a preset object storage medium according to the data update notification information;

A receiving information sending module is used to send data reception information to the middle station system when the data to be processed is received;

An effective information receiving module is used to receive data effective notification information;

A data validation module, used for validating all the received data to be processed according to the data validation notification information;

A data table updating module is used to update the pre-processed data table if all the pre-processed data in the pre-processed data set have taken effect. Suppose the data version management table in the database is updated.
An electronic device, comprising:

one or more processors;

Memory; and

One or more application programs, wherein the one or more application programs are stored in the memory and are configured to be executed by the processor to implement the data processing method according to any one of claims 1 to 6 or 7 to 12.
A computer-readable storage medium storing a computer program for executing the data processing method according to any one of claims 1 to 6 or 7 to 12.