CN115600560B

CN115600560B - Data conversion method, device and system, electronic equipment and storage medium

Info

Publication number: CN115600560B
Application number: CN202211190491.1A
Authority: CN
Inventors: 赵庆超
Original assignee: Zhongdian Jinxin Software Co Ltd
Current assignee: Zhongdian Jinxin Software Co Ltd
Priority date: 2022-09-28
Filing date: 2022-09-28
Publication date: 2023-06-20
Anticipated expiration: 2042-09-28
Also published as: CN115600560A

Abstract

The application relates to a data conversion method, a device, a system, an electronic device and a storage medium, wherein the method comprises the following steps: the data exchange layer receives a plurality of first data texts sent by a first service system; the data access layer reads the data to be converted in the plurality of first data texts from the data exchange layer and stores the data to be converted in the plurality of first data texts; the data conversion layer acquires data to be converted in a plurality of first data texts from the data access layer; the data conversion layer converts the data to be converted in the plurality of first data texts to obtain a plurality of second data texts; the data issuing layer reads the converted data in the plurality of second data texts from the data conversion layer and sends the converted data in the plurality of second data texts to the second service system. The data conversion system improves the performance of the data conversion system by processing the data to be converted in the plurality of first data texts in parallel and exporting the result set in the form of a plurality of second data texts.

Description

Data conversion method, device and system, electronic equipment and storage medium

Technical Field

The present disclosure relates to the field of data processing technologies, and in particular, to a data conversion method, apparatus, system, electronic device, and storage medium.

Background

The data conversion system belongs to a subsystem of the database management system, and can realize data storage, data conversion and data circulation. Specifically, the data conversion system may store data to be converted; and carrying out data conversion on the stored data to be converted, and storing the converted data into a corresponding database.

In practical applications, the data conversion system is generally required to support large-scale data upstream, and in related technologies, when the data conversion system performs data storage, data conversion and data circulation on large-scale data upstream, there is a problem of low efficiency.

Disclosure of Invention

In order to solve the above technical problems in the related art, embodiments of the present application provide a data conversion method, device, system, electronic device, and storage medium:

in a first aspect, an embodiment of the present application provides a data conversion method, which is applied to a data conversion system, where the data conversion system includes a data exchange layer, a data access layer, a data conversion layer, and a data distribution layer, and the method includes:

The data exchange layer receives a plurality of first data texts sent by a first service system; each first data text stores data to be converted;

the data access layer reads the data to be converted in the plurality of first data texts from the data exchange layer and stores the data to be converted in the plurality of first data texts;

the data conversion layer acquires data to be converted in the plurality of first data texts from the data access layer;

the data conversion layer converts the data to be converted in the plurality of first data texts to obtain a plurality of second data texts; each first data text corresponds to one second data text, and converted data is stored in each second data text;

the data issuing layer reads the converted data in the plurality of second data texts from the data conversion layer and sends the converted data in the plurality of second data texts to a second service system.

Optionally, the method further comprises:

the data exchange layer receives a plurality of first data volume texts sent by the first service system; each first data text corresponds to a first data volume text, and the data volume size of the first data text is stored in the first data volume text;

The data access layer reads the plurality of first data texts and stores data to be converted stored in the plurality of first data texts, and the data access layer comprises:

for each first data text, the data access layer detects whether a first data volume text corresponding to the first data text exists or not;

when the data access layer detects that a first data amount text corresponding to the first data text exists, reading data to be converted from the first data text by calling a preset first function, and storing the data to be converted into a target database; the target data block is a database in the data conversion system.

Optionally, the data conversion layer converts the data to be converted in the plurality of first data texts to obtain a plurality of second data texts, including:

for each first data text, the data conversion layer converts the data to be converted stored in the first data text based on a mapping relation between a first field of a pre-stored first data table and a second field of a second data table to obtain converted data; the first data table is a data table associated with a first service system, and the second data table is a data table associated with a second service system;

And the data conversion layer calls a preset second function to generate a plurality of second data texts from the converted data.

Optionally, the data issuing layer reads the converted data in the plurality of second data texts from the data converting layer, and sends the converted data in the plurality of second data texts to a second service system, where the data issuing layer includes:

the data issuing layer reads the converted data in the plurality of second data texts from the data conversion layer by calling a preset second function;

for each second data text, the data issuing layer generates a second data volume text based on the size of the data volume in the second data text; the second data volume text stores the data volume size of the second data text;

and the data issuing layer sends the plurality of second data texts and the generated plurality of second data volume texts to a second service system, so that when the second service system detects the second data volume texts, converted data in the second data texts corresponding to the second data volume texts are read.

Optionally, the data issuing layer sends the plurality of second data texts and the generated plurality of second data volume texts to a second service system, including:

The data issuing layer sends the plurality of second data texts and the generated plurality of second data volume texts to a target server so that the target server stores the plurality of second data texts and the plurality of second data volume texts, and the second business system reads the converted data in the plurality of second data texts from the target server and stores the converted data stored in the plurality of second data texts into a second data table.

Optionally, the data conversion system further comprises a data storage layer; the method further comprises the steps of:

the data storage layer acquires a predetermined target SQL script;

the data storage layer derives the plurality of first data texts from the data access layer and derives the plurality of second data texts from the data conversion layer by executing the target SQL script;

the data storage layer reads data to be converted in the plurality of first data texts and converted data in the plurality of second data texts by calling the first function;

the data storage layer stores data to be converted in the plurality of first data texts and converted data in the plurality of second data texts.

In a second aspect, an embodiment of the present application provides a data conversion device, which is applied to a data conversion system, where the data conversion system includes a data exchange layer, a data access layer, a data conversion layer, and a data issuing layer, and the device includes:

the data text receiving module is used for triggering the data exchange layer to receive a plurality of first data texts sent by the first service system; each first data text stores data to be converted;

the first data reading module is used for triggering the data access layer to read the data to be converted in the plurality of first data texts from the data exchange layer and storing the data to be converted in the plurality of first data texts;

the second data reading module is used for triggering the data conversion layer to acquire data to be converted in the plurality of first data texts from the data access layer;

the data conversion module is used for triggering the data conversion layer to convert the data to be converted in the plurality of first data texts to obtain a plurality of second data texts; each first data text corresponds to one second data text, and converted data is stored in each second data text;

The data sending module is used for triggering the data issuing layer to read the converted data in the plurality of second data texts from the data conversion layer and sending the converted data in the plurality of second data texts to a second service system.

Optionally, the apparatus further includes:

the data volume text receiving module is used for triggering the data exchange layer to receive a plurality of first data volume texts sent by the first service system; each first data text corresponds to a first data volume text, and the data volume size of the first data text is stored in the first data volume text;

the first data reading module is specifically configured to:

triggering the data access layer to detect whether a first data volume text corresponding to the first data text exists or not for each first data text;

Optionally, the data conversion module is specifically configured to:

for each first data text, triggering the data conversion layer to convert the data to be converted stored in the first data text based on a mapping relation between a first field of a pre-stored first data table and a second field of a second data table to obtain converted data; the first data table is a data table associated with a first service system, and the second data table is a data table associated with a second service system;

Optionally, the data sending module is specifically configured to:

triggering the data issuing layer to read converted data in the plurality of second data texts from the data conversion layer by calling a preset second function;

Optionally, the data sending module is further specifically configured to:

Optionally, the data conversion system further comprises a data storage layer; the apparatus further comprises:

the script determining module is used for triggering the data storage layer to acquire a predetermined target SQL script;

a data text deriving module, configured to trigger the data storage layer to derive the plurality of first data texts from the data access layer and derive the plurality of second data texts from the data conversion layer by executing the target SQL script;

the third data reading module is used for triggering the data storage layer to read the data to be converted in the plurality of first data texts by calling the first function and the converted data in the plurality of second data texts;

And the data storage module is used for triggering the data storage layer to store the data to be converted in the plurality of first data texts and the converted data in the plurality of second data texts.

In a third aspect, an embodiment of the present application provides a data conversion system, including a data exchange layer, a data access layer, a data conversion layer, and a data distribution layer;

the data exchange layer is used for receiving a plurality of first data texts sent by the first service system; each first data text stores data to be converted;

the data access layer is used for reading the data to be converted in the plurality of first data texts from the data exchange layer and storing the data to be converted in the plurality of first data texts;

the data conversion layer is used for acquiring data to be converted in the plurality of first data texts from the data access layer;

the data conversion layer is further used for converting data to be converted in the plurality of first data texts to obtain a plurality of second data texts; each first data text corresponds to one second data text, and converted data is stored in each second data text;

the data issuing layer is used for reading the converted data in the plurality of second data texts from the data conversion layer and sending the converted data in the plurality of second data texts to a second service system.

Optionally, the data exchange layer is further configured to receive a plurality of first data volume texts sent by the first service system; each first data text corresponds to a first data volume text, and the data volume size of the first data text is stored in the first data volume text;

for each first data text, the data access layer is specifically configured to detect whether a first data volume text corresponding to the first data text exists;

Optionally, for each first data text, the data conversion layer is specifically configured to convert data to be converted stored in the first data text based on a mapping relationship between a first field of a pre-stored first data table and a second field of a second data table, so as to obtain converted data; the first data table is a data table associated with a first service system, and the second data table is a data table associated with a second service system;

The data conversion layer is specifically used for calling a preset second function to generate a plurality of second data texts from the converted data.

Optionally, the data issuing layer is specifically configured to read, by calling a second function set in advance, converted data in the plurality of second data texts from the data conversion layer;

for each second data text, the data issuing layer is specifically configured to generate a second data text based on the size of the data amount in the second data text; the second data volume text stores the data volume size of the second data text;

the data issuing layer is specifically configured to send the plurality of second data texts and the generated plurality of second data volume texts to a second service system, so that when the second service system detects the second data volume text, the second service system reads converted data in the second data text corresponding to the second data volume text.

Optionally, the data issuing layer is specifically configured to send the plurality of second data texts and the generated plurality of second data amount texts to a target server, so that the target server stores the plurality of second data texts and the plurality of second data amount texts, and the second service system reads the converted data in the plurality of second data texts from the target server, and stores the converted data stored in the plurality of second data texts into a second data table.

Optionally, the data storage layer is configured to obtain a predetermined target SQL script;

the data storage layer is further configured to derive the plurality of first data texts from the data access layer and the plurality of second data texts from the data conversion layer by executing the target SQL script;

the data storage layer is further used for reading data to be converted in the plurality of first data texts by calling the first function, and converted data in the plurality of second data texts;

the data storage layer is further used for storing data to be converted in the first data texts and converted data in the second data texts.

In a fourth aspect, an embodiment of the present application provides an electronic device, including:

a processor;

a memory for storing the processor-executable instructions;

wherein the processor is configured to execute the instructions to implement the data conversion method of the first aspect.

In a fifth aspect, embodiments of the present application provide a storage medium, which when executed by a processor of an electronic device, enables the electronic device to perform the data conversion method of the first aspect.

In a sixth aspect, embodiments of the present application provide a computer program product, which when run on a computer, causes the computer to implement the data conversion method of the first aspect.

The data conversion method is applied to a data conversion system, when a large amount of data to be converted needs to be converted by the data conversion system, the first service system sends the data to be converted to the data exchange layer in the form of data texts, the data conversion system receives a plurality of first data texts, and the data access layer loads and stores the data to be converted in the plurality of first data texts in the form of reading the data texts; the data conversion layer acquires data to be converted in a plurality of first data texts from the data access layer, converts the data to be converted in the plurality of first data texts in parallel, exports a result set in a plurality of second data texts, and the data issuing layer loads the converted data in a form of reading the second data texts and sends the converted data in the plurality of second data texts to the second service system.

According to the technical scheme provided by the embodiment of the application, the data conversion system processes the data to be converted in the plurality of first data texts in parallel, and the result set is exported in the form of the plurality of second data texts, so that the processing speeds of data storage, data conversion and data circulation can be improved, the efficiency of data storage, data conversion and data circulation is further improved, and the performance of the data conversion system is improved.

Drawings

FIG. 1 is a schematic diagram illustrating interactions of a data exchange layer, a data access layer, a data conversion layer, a data delivery layer, and a data storage layer in a data conversion system according to an exemplary embodiment;

FIG. 2 is a flowchart of a data conversion method according to an embodiment of the present application;

FIG. 3 is a flow chart of one implementation of step S220 in the example of FIG. 2;

FIG. 4 is a flow chart of one implementation of step S240 in the example of FIG. 2;

FIG. 5 is a flow chart of one implementation of step S250 in the example of FIG. 2;

FIG. 6 is a flowchart of steps performed by the data storage layer;

FIG. 7 is a block diagram of a data conversion device according to an embodiment of the present application;

FIG. 8 is a block diagram of a data conversion system provided by an embodiment of the present application;

fig. 9 is a block diagram of an electronic device provided in an embodiment of the present application.

Detailed Description

In order to enable those skilled in the art to better understand the technical solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

It should be noted that the terms "first," "second," and the like in the description and claims of the present application and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that embodiments of the present application described herein may be implemented in sequences other than those illustrated or otherwise described herein. The implementations described in the following exemplary examples are not representative of all implementations consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present application as detailed in the accompanying claims.

In order to solve the above technical problems in the related art, embodiments of the present application provide a data conversion method, apparatus, electronic device, and storage medium.

The data conversion method provided by the embodiment of the application can be applied to a data conversion system. For clarity of solution description, the technical solution of the embodiments of the present disclosure will be described in detail with reference to the architecture of the data conversion system in the first aspect.

As shown in fig. 1, the data conversion system according to the embodiment of the present application may include a data exchange layer 110, a data access layer 120, a data conversion layer 130, a data delivery layer 140, and a data storage layer 150, where an interaction procedure between the data exchange layer 110, the data access layer 120, the data conversion layer 130, the data delivery layer 140, and the data storage layer 150 is as follows:

The first service system is associated with a plurality of data tables, each data table is usually stored with a large amount of data, and when the data in the data table associated with the first service system needs to be converted and stored into the data table of the second service system, the first service system sends a plurality of data texts to the data conversion system, and the plurality of data texts are called as first data texts for the sake of clarity of scheme description.

Specifically, for each data table associated with the first service system, the first service system generates two texts, namely a first data text and a first data volume text, wherein the first data text stores data to be converted, and the first data volume text stores the data volume of the data in the first data text. And, the first service system may transmit the first data text and the first data amount text to the data exchange layer 110 of the data conversion system, so that the data exchange layer 110 may receive the plurality of first data texts and the plurality of first data amount texts.

The data access layer 120 detects whether the first data amount text exists in the data exchange layer 110, in one case, if the data access layer 120 detects that the first data amount text exists in the data exchange layer 110, it is determined that the data to be converted in the first data text is normal, that is, the data is not abnormal, and loading of the data by reading the form of the first data text is started. Since there are a plurality of first data texts, the efficiency to be converted in the first data texts can be read through multithreading, so that the data reading efficiency is improved. Specifically, the data access layer 120 reads the data to be converted in the first data text by calling a preset first function, such as an import function, and stores the read data to be converted in a database in the data conversion system.

In another case, if the data access layer 120 does not detect that the first data amount text exists in the data exchange layer 110, it is determined that the data to be converted in the first data text is abnormal, and a data error is reported, and loading of the data fails; and retrying loading the data to be converted in the first data text, if retrying is performed for N times, for example, N can be equal to 3, and the data loading is still wrong, the problem of the data can be solved by a mode of participation of a user.

The data to be converted in the first data text may be full data or incremental data, the full data may be data generated in a longer period of time before the current time, and the incremental data may be data generated in a shorter period of time before the current time. For example, the delta data may include only data generated on the same day; the full volume data may include data generated some time before the day.

In practical application, only the data to be converted received at this time can be reserved in a database in the data conversion system, and the data to be converted received before can be deleted, so that the data storage performance, the data conversion performance and the data storage performance are improved.

The data conversion layer 130 stores a first data dictionary corresponding to the first service system and a second data dictionary corresponding to the second service system, where a first field in a first data table associated with the first service system may be stored in the first data dictionary, and a mapping relationship between a second field in a second data table associated with the second service system, that is, what value in the second data table is stored in the first field in the first data table corresponds to. Likewise, the second data dictionary may store a second field in a second data table associated with the second service system, and a mapping relationship between the second field in the first data table associated with the first service system, that is, what value of the first data table is stored in correspondence with the second field in the second data table. By storing the first data dictionary and the second data dictionary, the capability of converting the data of the first service system into the data of the second service system, namely, the service mapping, and carrying the service data conversion can be realized.

The data conversion layer 130 may specifically perform two part operations, where the first part operation is to read data to be converted in the plurality of first data texts from the data access layer through a pre-developed SQL script, and convert the data to be converted in the plurality of first data texts in parallel. The second part is operated to generate a plurality of second data texts through a preset second function such as an Export function, wherein each first data text corresponds to one second data text, and converted data is stored in each second data text. And, the data conversion layer 120 may combine a plurality of second data texts into one second data text.

The data issuing layer 140 may read the second data text from the data converting layer 130 and generate a second data amount text based on the data amount size of the converted data in the second data text. And issuing the second data text and the second data amount text to a destination server, such as a ferry. The second business system may read the second data text and the second data amount text from the ferry and store the second data text in a database associated therewith.

In order to ensure timeliness of batch data conversion, after the data conversion layer 130 performs data conversion and the data transmission layer 140 performs data transmission, the data storage layer 150 acquires a predetermined SQL script, and by executing the SQL script, derives a first data text from the data access layer 120 and derives a plurality of second data texts from the data conversion layer 130; and reading the data to be converted in the first data text and the converted data in the second data text by calling a first function such as a digital import function, and storing the data to be converted in the plurality of first data texts and the converted data in the plurality of second data texts.

By storing and archiving data in the data table associated with the first business system (pre-conversion data) and converted data, post-SIT ((System Integration Testing, system integration test), UAT (User Acceptance Test ) and business data conversion query retrospection are facilitated and, in order to achieve accurate storage of data, the data interface of the data storage layer 150 is consistent with the data interfaces of the first business system and the second business system.

It can be seen that the data conversion method provided in the embodiments of the present application is applied to a data conversion system, where the data conversion system includes a data exchange layer, a data access layer, a data conversion layer, a data distribution layer, and a data storage layer. When a large amount of data to be converted needs to be converted by the data conversion system, the first service system sends the data to be converted to the data exchange layer in the form of data texts, the data conversion system receives a plurality of first data texts, and the data access layer loads and stores the data to be converted in the plurality of first data texts in the form of reading the data texts; the data conversion layer acquires data to be converted in a plurality of first data texts from the data access layer, converts the data to be converted in the plurality of first data texts in parallel, exports a result set in a plurality of second data texts, and the data issuing layer loads the converted data in a form of reading the second data texts and sends the converted data in the plurality of second data texts to the second service system.

According to the technical scheme provided by the embodiment of the application, the processing speeds of data storage, data conversion and data circulation can be improved by processing the data to be converted in the plurality of first data texts in parallel, so that the efficiency of data storage, data conversion and data circulation is improved, and the performance of a data conversion system is improved.

After the technical solution of the embodiments of the present application is described in detail with reference to the architecture of the data conversion system, in a second aspect, a data conversion method provided by the embodiments of the present application will be described in detail. The data conversion method can be applied to the data conversion method described in the embodiment shown in fig. 1.

As shown in fig. 2, a data conversion method provided in the embodiment of the present application may include the following steps:

s210, the data exchange layer receives a plurality of first data texts sent by the first service system.

Wherein each first data text stores data to be converted.

In particular, the first service system is associated with a plurality of data tables, each data table usually stores a large amount of data, when the data in the data table associated with the first service system needs to be converted and stored into the data table of the second service system, the first service system sends a plurality of data texts to the data conversion system, for the sake of clarity of description of the scheme, the plurality of data texts are called as first data texts, so that the data exchange layer receives the plurality of first data texts sent by the first service system,

s220, the data access layer reads the data to be converted in the plurality of first data texts from the data exchange layer and stores the data to be converted in the plurality of first data texts.

Specifically, after the data exchange layer receives a plurality of first data texts sent by the first service system, the data access layer starts to load data, that is, the data access layer reads data to be converted in the plurality of first data texts from the data exchange layer. Since there are a plurality of first data texts, the efficiency to be converted in the first data texts can be read through multithreading, so that the data reading efficiency is improved.

In practical application, the data access layer may read the data to be converted in the first data text through a preset first function, such as an import function, and store the read data to be converted in a database in the data conversion system.

In addition, only the data to be converted received at this time can be reserved in a database in the data conversion system, and the data to be converted received before can be deleted, so that the data storage performance, the data conversion performance and the data storage performance are improved.

And S230, the data conversion layer acquires data to be converted in a plurality of first data texts from the data access layer.

Specifically, the data conversion layer may read data to be converted in the plurality of first data texts from the data access layer through a pre-developed SQL script.

S240, the data conversion layer converts the data to be converted in the plurality of first data texts to obtain a plurality of second data texts.

Each first data text corresponds to one second data text, and converted data is stored in each second data text.

Specifically, after the data conversion layer obtains the data to be converted in the plurality of first data texts from the data access layer, the data to be converted in the plurality of first data texts are converted in parallel. And generating a plurality of second data texts through a preset second function such as an Export function, wherein each first data text corresponds to one second data text, and converted data is stored in each second data text. And, the data conversion layer may combine the plurality of second data texts into one second data text.

For clarity of description of the solution, a specific implementation of S240 will be explained in detail in the following examples.

S250, the data issuing layer reads the converted data in the plurality of second data texts from the data conversion layer and sends the converted data in the plurality of second data texts to the second service system.

Specifically, after the data conversion layer performs data conversion to obtain a plurality of second data texts, the data issuing layer may read the converted data in the plurality of second data texts from the data conversion layer in a manner of reading the second data texts, and send the converted data in the plurality of second data texts to the second service system.

Based on the example shown in fig. 2, in one implementation manner, the data conversion method may further include the following step a:

and a step a, the data exchange layer receives a plurality of first data volume texts sent by the first service system.

Each first data text corresponds to a first data volume text, and the data volume size of the first data text is stored in the first data volume text.

Specifically, for each data table associated with the first service system, the first service system may generate two texts, which are a first data text and a first data volume text, where data to be converted is stored in the first data text, and a data volume of data in the first data text is stored in the first data volume text. And, the first business system may send the first data text and the first data amount text to a data exchange layer of the data conversion system, so that the data exchange layer may receive the plurality of first data texts and the plurality of first data amount texts.

At this time, S220, the data access layer reads the plurality of first data texts and stores the data to be converted stored in the plurality of first data texts, as shown in fig. 3, may include the following steps:

s221, for each first data text, the data access layer detects whether a first data volume text corresponding to the first data text exists.

S222, when the data access layer detects that a first data amount text corresponding to the first data text exists, the data to be converted is read from the first data text by calling a preset first function, and the data to be converted is stored in the target database.

Wherein the target data block is a database in the data conversion system.

The data access layer detects whether the first data amount text exists in the data exchange layer, and if the data access layer detects that the first data amount text exists in the data exchange layer, the data access layer starts to load data in the form of reading the first data text. Since there are a plurality of first data texts, the efficiency to be converted in the first data texts can be read through multithreading, so that the data reading efficiency is improved. Specifically, the data access layer may read the data to be converted in the first data text by calling a preset first function, such as an import function, and store the read data to be converted in a database in the data conversion system.

If the data access layer does not detect that the first data amount text exists in the data exchange layer, the data to be converted in the first data text is possibly abnormal, so that data error reporting is carried out, and data loading is failed; and retrying loading the data to be converted in the first data text, if retrying is performed for N times, for example, N can be equal to 3, and the data loading is still wrong, the problem of the data can be solved by a mode of participation of a user.

According to the technical scheme provided by the embodiment, before the data to be converted is loaded from the data exchange layer, the data access layer can detect whether the first data amount text exists in the data exchange layer, and when the first data amount text exists in the data exchange layer, the data to be converted is loaded in a form of reading the first data text, so that the accuracy of the data loaded by the data access layer is guaranteed.

Based on the above embodiment, in one implementation manner, S240, the data conversion layer converts the data to be converted in the plurality of first data texts to obtain a plurality of second data texts, as shown in fig. 4, may include the following steps:

s241, for each first data text, the data conversion layer converts the data to be converted stored in the first data text based on the mapping relation between the first field of the first data table and the second field of the second data table stored in advance, and the converted data is obtained.

The first data table is a data table associated with the first service system, and the second data table is a data table associated with the second service system.

Specifically, the data conversion layer may store a first data dictionary corresponding to the first service system and a second data dictionary corresponding to the second service system, where the first data dictionary may store a first field in a first data table associated with the first service system and a mapping relationship between a second field in a second data table associated with the second service system, that is, what value of the second data table is stored in the first field in the first data table corresponds to. Likewise, the second data dictionary may store a second field in a second data table associated with the second service system, and a mapping relationship between the second field in the first data table associated with the first service system, that is, what value of the first data table is stored in correspondence with the second field in the second data table. By storing the first data dictionary and the second data dictionary, the capability of converting the data of the first service system into the data of the second service system, namely, the service mapping, and carrying the service data conversion can be realized.

For each first data text, the data conversion layer can convert the data to be converted stored in the first data text based on the mapping relation between the first field of the first data table and the second field of the second data table, which are stored in advance, so as to accurately obtain the converted data. Specifically, the data to be converted in the plurality of first data texts can be read from the data access layer through the pre-developed SQL script, and the data to be converted in the plurality of first data texts are converted in parallel.

S242, the data conversion layer calls a preset second function to generate a plurality of second data texts from the converted data.

Specifically, the data conversion layer may generate a plurality of second data texts by calling a preset second function, such as an Export function, where each first data text corresponds to one second data text, and each second data text stores converted data. And, the data conversion layer may combine the plurality of second data texts into one second data text.

According to the technical scheme provided by the embodiment, the data conversion layer converts the data to be converted stored in the first data text based on the mapping relation between the first field of the first data table and the second field of the second data table, which are stored in advance, so that the converted data can be accurately obtained. And the result set is exported in the form of a plurality of second data texts, so that the processing speed of data storage, data conversion and data circulation can be improved, the efficiency of data storage, data conversion and data circulation is further improved, and the performance of the data conversion system is improved.

Based on the foregoing embodiment, in one implementation, S250, the data issuing layer reads the converted data in the plurality of second data texts from the data converting layer and sends the converted data in the plurality of second data texts to the second service system, as shown in fig. 5, may include the following steps:

s251, the data issuing layer reads the converted data in the plurality of second data texts from the data conversion layer by calling a preset second function.

The data issuing layer may obtain the converted data in the second data text from the data converting layer in the form of reading the data text by calling a second function, such as an Export function, which is preset.

S252, for each second data text, the data issuing layer generates a second data text based on the size of the data amount in the second data text.

Wherein, the second data volume text stores the data volume size of the second data text.

Specifically, the data issuing layer may read the second data text from the data converting layer, and generate the second data volume text based on the data volume size of the converted data in the second data text.

And S253, the data issuing layer sends a plurality of second data texts and the generated plurality of second data volume texts to the second service system, so that when the second service system detects the second data volume texts, the second service system reads the converted data in the second data texts corresponding to the second data volume texts.

Specifically, the second data text and the second data volume text are issued to the second service system, and for each second data text, the second service system can detect whether the second data volume text corresponding to the second data text exists, and when the second data volume text exists in the second service system, the second service system starts to read the converted data in the second data text and stores the second data text in a data table associated with the second data text.

Therefore, according to the technical scheme provided by the embodiment, the data issuing layer generates a plurality of second data texts and a plurality of second data volume texts and sends the plurality of second data texts and the plurality of second data volume texts to the second service system, when the second data volume texts exist in the second service system, the second service system starts to read the converted data in the second data texts and stores the second data texts in the database associated with the second service system, so that the second service system can accurately store the converted data in the data table associated with the second service system.

As an implementation manner of the embodiment of the present application, the data issuing layer sends a plurality of second data texts and a plurality of generated second data volume texts to the second service system, and may include the following step b:

And b, the data issuing layer sends a plurality of second data texts and the generated plurality of second data volume texts to the target server so that the target server stores the plurality of second data texts and the plurality of second data volume texts, and the second service system reads the converted data in the plurality of second data texts from the target server and stores the converted data stored in the plurality of second data texts into a second data table.

Specifically, the data issuing layer may issue the plurality of second data texts and the plurality of second data volume texts to a target server in communication with the second service system, where the target server may be a ferry machine, and the target server may function as a hub, and when the second service system needs to read the converted data in the second data text, the second service system may obtain the converted data from the ferry machine in a form of reading the second data text with higher efficiency.

Based on the foregoing embodiment, in one implementation, the data conversion system further includes a data storage layer; as shown in fig. 6, the data conversion method may further include the steps of:

s610, the data storage layer acquires a predetermined target SQL script.

S620, the data storage layer derives a plurality of first data texts from the data access layer and derives a plurality of second data texts from the data conversion layer by executing the target SQL script.

S630, the data storage layer reads the data to be converted in the plurality of first data texts and the converted data in the plurality of second data texts by calling the first function.

S640, the data storage layer stores the data to be converted in the plurality of first data texts and the converted data in the plurality of second data texts.

In order to ensure timeliness of batch data conversion, the data conversion layer performs data conversion, the data storage layer acquires a predetermined script after the data transmission layer performs data transmission, and the SQL script is executed to lead out a first data text from the data access layer and lead out a plurality of second data texts from the data conversion layer; and reading the data to be converted in the first data text and the converted data in the second data text by calling a first function such as a digital import function, and storing the data to be converted in the plurality of first data texts and the converted data in the plurality of second data texts.

According to the technical scheme provided by the embodiment, the data (the data before conversion) in the data table related to the first service system and the converted data are stored and archived, so that later SIT ((System Integration Testing, system integration test), UAT (User Acceptance Test ) and data conversion query tracing are facilitated.

For clarity of description of the solution, the following detailed description of the embodiments of the present application will be presented in conjunction with the specific embodiments.

When the first service system has data to be converted, a user can configure a dispatching platform job, the time automatically triggers a virtual father job, and a first data text and a first data volume text are sent to a data exchange layer.

The second step is that the virtual parent initiates the completion of the job, triggering the data loading layer (i.e., the data access layer as described in the above embodiments) job.

And thirdly, operating the data access layer. The text of the corresponding first data amount is automatically detected under the specified directory, which may be called an OK file, if the OK file exists, the first data file is detected, which may be called a txt file, if the txt file exists, the data is loaded. If the OK file and the txt file do not exist and the data has problems, the operation fails to report errors, the data loading fails, and the error reporting is performed after retrying for 3 times, so that the operation is manually participated. The data storage mode may be to store only data generated on the same day.

And step four, loading data by the data access layer. The data to be converted in the first data text may be full data, or may be incremental data, where the full data may be data generated in a longer period of time before the current time, and the incremental data may be data generated in a shorter period of time before the current time. For example, the delta data may include only data generated on the same day; the full volume data may include data generated some time before the day. When the data to be converted in the first data text is full data, the data loading layer loads the full data; when the data to be converted in the first data text is incremental data, the data loading layer loads the incremental data.

And fifthly, operating the data conversion layer. The data conversion layer converts the data to be converted to obtain a second data text, and the converted data is stored in the second data text.

And a sixth step, the data conversion layer sends the converted data in the second data text to the data issuing layer.

Seventh step, data issuing layer operation. The data issuing layer automatically uploads the second data text and a verification file such as an OK file to the ferry.

Eighth step, data storage layer operation. The data conversion layer transmits the converted data to the data storage layer; the data access layer transmits the data to be converted to the data storage layer, and the data storage layer stores the converted data and the data to be converted. And when the data conversion inquiry is traced, the data to be converted stored in the data storage layer can be restored to the data access layer, and the converted data stored in the data storage layer can also be restored to the data conversion layer.

It should be noted that, in the above embodiment, the above steps have been described in detail, and are not described herein again.

In a second aspect, an embodiment of the present application provides a data conversion device, which is applied to a data conversion system, where the data conversion system includes a data exchange layer, a data access layer, a data conversion layer, and a data distribution layer, as shown in fig. 7, and the device includes:

A data text receiving module 710, configured to trigger the data exchange layer to receive a plurality of first data texts sent by the first service system; each first data text stores data to be converted;

a first data reading module 720, configured to trigger the data access layer to read data to be converted in the plurality of first data texts from the data exchange layer, and store the data to be converted in the plurality of first data texts;

a second data reading module 730, configured to trigger the data conversion layer to obtain data to be converted in the plurality of first data texts from the data access layer;

the data conversion module 740 is configured to trigger the data conversion layer to convert the data to be converted in the plurality of first data texts, so as to obtain a plurality of second data texts; each first data text corresponds to one second data text, and converted data is stored in each second data text;

and the data sending module 750 is configured to trigger the data issuing layer to read the converted data in the plurality of second data texts from the data conversion layer, and send the converted data in the plurality of second data texts to a second service system.

Optionally, the apparatus further includes:

the first data reading module is specifically configured to:

Optionally, the data conversion module is specifically configured to:

Optionally, the data sending module is specifically configured to:

Optionally, the data sending module is further specifically configured to:

In a third aspect, an embodiment of the present application provides a data conversion system, as shown in fig. 8, including a data exchange layer 810, a data access layer 820, a data conversion layer 830, and a data distribution layer 840;

In a fourth aspect, an embodiment of the present application provides an electronic device, as shown in fig. 9, including:

a processor 910;

a memory 920 for storing the processor-executable instructions;

Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the application following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the application pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the following claims.

It is to be understood that the present application is not limited to the precise arrangements and instrumentalities shown in the drawings, which have been described above, and that various modifications and changes may be effected without departing from the scope thereof. The scope of the application is limited only by the appended claims.

Claims

1. A data conversion method, applied to a data conversion system, the data conversion system including a data exchange layer, a data access layer, a data conversion layer, and a data distribution layer, the method comprising:

the data issuing layer reads the converted data in the plurality of second data texts from the data conversion layer and sends the converted data in the plurality of second data texts to a second service system;

when the data access layer detects that a first data amount text corresponding to the first data text exists, reading data to be converted from the first data text by calling a preset first function, and storing the data to be converted into a target database; the target database is a database in the data conversion system; and only the data to be converted received at this time is reserved in the target database.

2. The method of claim 1, wherein the data conversion layer converts data to be converted in the plurality of first data texts to obtain a plurality of second data texts, including:

3. The method of claim 2, wherein the data delivery layer reads the converted data in the plurality of second data texts from the data conversion layer and sends the converted data in the plurality of second data texts to a second business system, comprising:

4. The method of claim 3, wherein the data delivery layer sending the plurality of second data texts and the generated plurality of second data amount texts to a second service system comprises:

5. The method of claim 1, wherein the data conversion system further comprises a data storage layer; the method further comprises the steps of:

The data storage layer acquires a predetermined target SQL script;

6. A data conversion device, characterized by being applied to a data conversion system, the data conversion system including a data exchange layer, a data access layer, a data conversion layer, and a data distribution layer, the device comprising:

the data sending module is used for triggering the data issuing layer to read the converted data in the plurality of second data texts from the data conversion layer and sending the converted data in the plurality of second data texts to a second service system;

7. The data conversion system is characterized by comprising a data exchange layer, a data access layer, a data conversion layer and a data issuing layer;

the data issuing layer is used for reading the converted data in the plurality of second data texts from the data conversion layer and sending the converted data in the plurality of second data texts to a second service system;

the data exchange layer is further used for receiving a plurality of first data volume texts sent by the first service system; each first data text corresponds to a first data volume text, and the data volume size of the first data text is stored in the first data volume text;

8. An electronic device, comprising:

a processor;

a memory for storing the processor-executable instructions;

wherein the processor is configured to execute the instructions to implement the data conversion method of any one of claims 1 to 5.

9. A storage medium, which when executed by a processor of an electronic device, causes the electronic device to perform the data conversion method of any one of claims 1 to 5.