CN106909570B - A data conversion method and device - Google Patents

Abstract

The present application discloses a data conversion method and device. The data conversion method is used to convert source data using different data models into target data using a unified demand data model. The method includes: obtaining the source data to be converted; reading the pre-stored conversion annotation of the source data corresponding to the data model The conversion annotation information at least includes: source data field information to be retained; according to the read conversion annotation information, the source data field to be retained is extracted from the source data to obtain target data. Compared with the prior art solution, the technical solution of the present application not only reduces the number of codes for implementing conversion logic, but also integrates multiple sets of data conversion logic that were originally scattered, thereby effectively reducing development and maintenance costs.

Description

A data conversion method and device

technical field

The present application relates to the technical field of data management, and in particular, to a data conversion method and device.

Background technique

Data management refers to the process of collecting, storing, processing and applying various kinds of data. In the era of big data, in order to cope with the significant increase in the amount of data, in addition to the need to provide sufficient storage space, data management technology has also been proposed. high demands.

For the data store or data user, it is often necessary to obtain data from an external data source, and when the data model provided by the external data source is inconsistent with the data model required for actual use, it needs to The acquired data is converted, and the content of the conversion may include the selection of fields, the change of field names, the change of field types, and so on.

In practical applications, various data models are obtained from different external data sources. According to the implementation scheme of the prior art, in order to meet unified data storage and usage requirements, it is necessary to write conversion logic for each external data model. In the case of a large number of data sources, a lot of development and maintenance costs need to be invested, and in many application scenarios, the number of data sources is constantly increasing, which also makes the overall development and maintenance costs difficult to control.

SUMMARY OF THE INVENTION

In view of the above-mentioned technical problems, the application provides a data conversion method and device, and the technical scheme is as follows:

A data conversion method for converting source data using different data models into target data using a unified demand data model, the method comprising:

Obtain the source data to be transformed;

Read the pre-stored conversion annotation information of the data model corresponding to the source data, where the conversion annotation information at least includes: source data field information to be retained;

According to the read conversion annotation information, the source data fields that need to be retained are extracted from the source data to obtain the target data.

A data conversion device for converting source data using different data models into target data using a unified demand data model, the device comprising:

The source data obtaining module is used to obtain the source data to be converted;

An annotation information reading module, configured to read the pre-stored conversion annotation information of the data model corresponding to the source data, where the conversion annotation information at least includes: source data field information to be retained;

The target data generation module is configured to extract the source data fields that need to be retained from the source data according to the read conversion annotation information to obtain the target data.

The technical solution provided by this application configures data conversion requirements for different source data models by annotating the source data models in advance. The logic of data conversion requirements can realize unified conversion of source data using different data models. Compared with the prior art solution, it not only reduces the number of implementation codes of conversion logic, but also integrates originally scattered sets of data conversion logic, thereby effectively reducing development and maintenance costs.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not limiting of the present application.

Description of drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only These are some embodiments described in this application. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings.

Fig. 1 is the schematic flow chart of the data conversion method of the present application;

Fig. 2 is the first structural representation of the data conversion device of the present application;

FIG. 3 is a schematic diagram of the second structure of the data conversion apparatus of the present application.

Detailed ways

In order to make those skilled in the art better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be described in detail below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only It is a part of the embodiments of the present application, but not all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application shall fall within the protection scope of the present application.

In order to solve the problem of separately developing conversion logic using multiple data models, the solution provided by this application is: according to application requirements, the fields of various source data models are marked in advance, and the marked content may include: which ones need to be retained field, whether to change the field name, whether to change the field type, etc. When the source data needs to be transformed, it is only necessary to invoke a set of general logic that can identify the above data transformation requirements, and then the source data using different data models can be transformed uniformly.

As shown in Fig. 1, the flow chart of the data conversion method that this application provides, this method may comprise the following steps:

S101, obtaining source data to be converted;

S102: Read the pre-stored conversion annotation information of the data model corresponding to the source data, where the conversion annotation information at least includes: source data field information to be retained;

S103, according to the read conversion annotation information, extract the source data fields that need to be retained from the source data to obtain target data.

The data conversion method provided by this application corresponds to the aforementioned general data conversion logic, and the above-mentioned data conversion method will be described below in conjunction with specific examples:

Field Name field description Field Type name Name String age age Long birthday Birthday Date height height Double

Table 1

Field Name field description Field Type name Name String age age Long birthday Birthday Date tel phone number String

Table 2

Table 1 and Table 2 respectively show the field information of two different source data models, wherein data model 1 includes four fields of name, age, birthday, height, and data model 2 includes name, age, birthday, tel four fields. In actual application requirements, only the data of the three fields of name, age, and birthday need to be used. Then, according to the solution of the present application, the fields of the data model 1 and the data model 2 can be marked in advance, respectively, and the conversion marking information of the two data models can be obtained as shown in Table 3 and Table 4 respectively:

Field Name field description Field Type callout name Name String 1 age age Long 1 birthday Birthday Date 1 height height Double 0

table 3

Field Name field description Field Type callout name Name String 1 age age Long 1 birthday Birthday Date 1 tel phone number String 0

Table 4

In Table 3 and Table 4, "0" and "1" respectively indicate whether the corresponding field is retained in the conversion result. It can be seen that the "height" field in Table 3 and the "tel" field in Table 4 are both It is marked as 0, indicating that the corresponding field does not need to be preserved in the conversion result. After the labeling is completed, the labeling information of each model is stored corresponding to the source data identifier.

In the transformation phase, it is assumed that the incoming source data based on data model 1 is shown in Table 5:

name age birthday height Zhang San 30 1980-12-12 1.87 Li Si 20 1990-01-12 1.70 Wang Wu 15 1995-08-12 1.60

table 5

According to the data conversion method provided by the present application, after obtaining the source data, first determine that the model used for the data is data model 1, and then read the conversion annotation information of data model 1. According to the conversion annotation information, it can be determined that the fields to be retained include name , age, birthday, after extracting the data of these three fields, the target data is obtained as shown in Table 6:

name age birthday Zhang San 30 1980-12-12 Li Si 20 1990-01-12 Wang Wu 15 1995-08-12

Table 6

In the above embodiment, the conversion annotation information is only used to identify whether each field of the source data needs to be retained in the conversion result, that is, no other conversion processing is required by default. In the actual conversion application, in addition to the selection processing of the fields of the source data, other conversion processing may also be required, such as field name conversion, field type conversion, etc. The above requirements are achieved by extending the transformation annotation information.

Assume that when converting data model 1, in addition to retaining the three fields of name, age, and birthday, in order to adapt to the demand data model, the names of the three fields of the target data to be converted are "username", "age", "birthday", the field types are string, short, and date respectively. It can be seen that the name of the original name field and the type of the original age field have all changed. The above requirements can be achieved through the annotation methods shown in Table 7.

Field Name field description Field Type callout name Name String @Define(targetName="username" targetType="String") age age Long @Define(targetName="age" targetType="Short") birthday Birthday Date @Define(targetName="birthday" targetType="Date") height height Double Null

Table 7

In Table 7, except for the field height that does not need to be reserved after conversion (marked with Null), the converted target field name targetName and the converted target field type targetType are specified for the three fields that need to be reserved. In actual data conversion, the fields that need to be preserved are extracted from the source data, and then the target data is generated according to the specified targetName and targetType.

It is understandable that in addition to specifying the target field name and target field type in the annotation information, other conversion requirements can also be specified, such as conversion of measurement units, rounding of numerical values, and even calling a specific function to the value of the source data field. After processing, those skilled in the art can expand the conversion annotation information according to actual needs. In addition, it should be noted that the information labeling method provided in this application is only for schematic illustration, and should not be construed as a limitation on the solution of this application.

In practical applications, the annotation of source data models can be achieved through simple configuration operations, and the benefits in exchange are: only a set of general transformation logic can be used to achieve unified transformation of source data using different data models , compared with the existing multiple sets of conversion logic, the development cost is significantly reduced; in addition, in the general conversion logic, not only a large number of redundant codes are reduced, but also multiple sets of data conversion logic that were originally scattered can be concentrated together, making The code is more readable and easier to maintain.

After the target data is generated, the generated data can be directly forwarded to the data application side, or the generated data can be stored. In a specific implementation manner provided by this application, in order to facilitate unified storage of data, target data may be saved in the form of single-valued data. That is, each row is single-valued data. In this single-valued data, it is necessary to reflect the key-value pair information of each field of the target data. Specifically, for each row of the target data, it is necessary to write the field names and corresponding values of the row into a string, and at the same time, it is necessary to provide a segmentation identifier in the string to distinguish each field, and this string is single-valued data. value of . Optionally, type information of the field can also be added to the single-valued data.

In practical applications, when performing a storage operation, you can directly use the target field name, target field type and other information in the conversion annotation information to give the value of the source data to obtain the data to be stored, for example, the three data shown in Table 8. , the first 4 columns are the source data based on the aforementioned data model 1. According to the label information shown in Table 7, the final result of the saved target data is shown in the last column of Table 8:

Table 8

With the above data storage method, data content of any type and any number of fields can be stored using only one content field, which is not only convenient for maintenance, but also effectively simplifies the data structure and improves the efficiency of data writing. When using the data subsequently, the specific field value can be restored directly according to the generation rules of the single-valued data.

Corresponding to the above method embodiments, the present application further provides a data conversion apparatus, as shown in FIG. 2 , the apparatus may include:

source data obtaining module 110, for obtaining source data to be converted;

The annotation information reading module 120 is configured to read the conversion annotation information of the data model corresponding to the pre-stored source data, wherein the conversion annotation information at least includes: source data field information to be retained;

The target data generation module 130 is configured to extract the source data fields that need to be retained from the source data according to the read conversion annotation information to obtain the target data.

In a specific embodiment of the present application, the conversion annotation information may further include: the converted target field name of the source data field that needs to be retained;

Correspondingly, the target data generation module 130 may be specifically configured to: extract the source data fields that need to be retained from the source data according to the read conversion annotation information, and convert the field names into corresponding target field names to obtain the target data.

In a specific embodiment of the present application, the conversion annotation information may further include: the converted target field type of the source data field that needs to be retained;

Correspondingly, the target data generation module 130 may be specifically configured to: extract the source data fields that need to be retained from the source data according to the read conversion annotation information, and convert the field types to the corresponding target field types to obtain the target data.

Referring to FIG. 3 , in a specific embodiment of the present application, the above-mentioned data conversion apparatus may further include a saving module 140 for saving target data as single-valued data, and the value of the single-valued data carries the target data Key-value pair information for each field.

Optionally, the value of the single-valued data may also carry field type information of each field of the target data.

For details of the implementation process of the functions and functions of each module in the above-mentioned device, please refer to the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

From the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solutions of the present application can be embodied in the form of software products in essence or the parts that make contributions to the prior art, and the computer software products can be stored in storage media, such as ROM/RAM, magnetic disks , CD-ROM, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present application.

The various embodiments in this specification are described in a progressive manner, and the same and similar parts between the various embodiments may be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the apparatus embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and reference may be made to some descriptions of the method embodiments for related parts. The device embodiments described above are only illustrative, and the modules described as separate components may or may not be physically separated. When implementing the solution of the present application, the functions of each module may be integrated into one or more modules. implemented in software and/or hardware. Some or all of the modules may also be selected according to actual needs to achieve the purpose of the solution in this embodiment. Those of ordinary skill in the art can understand and implement it without creative effort.

The above are only specific embodiments of the present application. It should be pointed out that for those skilled in the art, without departing from the principles of the present application, several improvements and modifications can also be made. It should be regarded as the protection scope of this application.

Claims (10)

1. A data conversion method, applied to the data collector, used to call a set of general logic to convert source data using different data models into target data using a unified demand data model, and pre-store different data corresponding to different data models. Converting the annotation information to convert different data models into the same data model, characterized in that the method includes: Obtain the source data to be transformed; Reading the pre-stored conversion annotation information of the data model corresponding to the source data, the conversion annotation information at least includes: the source data field name and field type to be retained; According to the read conversion annotation information, the source data fields that need to be retained are extracted from the source data to obtain the target data. 2. The method according to claim 1, wherein the conversion annotation information further comprises: the target field name after the conversion of the source data field that needs to be retained; Extracting the source data fields that need to be retained from the source data according to the read conversion annotation information to obtain the target data includes: according to the read conversion annotation information, extracting from the source data the fields that need to be retained. source data field, and convert the field name to the corresponding target field name to obtain the target data. 3. The method according to claim 1, wherein the conversion annotation information further comprises: the target field type after the conversion of the source data field that needs to be retained; Extracting the source data fields that need to be retained from the source data according to the read conversion annotation information to obtain the target data includes: according to the read conversion annotation information, extracting from the source data the fields that need to be retained. The source data field, and the field type is converted to the corresponding target field type to obtain the target data. 4. The method according to claim 1, wherein the method further comprises: The target data is saved as single-valued data, and the value of the single-valued data carries key-value pair information of each field of the target data. 5 . The method according to claim 4 , wherein the value of the single-valued data further carries field type information of each field of the target data. 6 . 6. A data conversion device, applied to a data collector, used to call a set of general logic to convert source data using different data models into target data using a unified demand data model, and pre-store different data corresponding to different data models. Converting annotation information to convert different data models into the same data model, characterized in that the device includes: The source data obtaining module is used to obtain the source data to be converted; An annotation information reading module, configured to read the pre-stored conversion annotation information of the data model corresponding to the source data, where the conversion annotation information at least includes: source data field names and field types to be retained; The target data generation module is configured to extract the source data fields that need to be retained from the source data according to the read conversion annotation information to obtain the target data. 7. The device according to claim 6, wherein the conversion annotation information further comprises: the target field name after the conversion of the source data field that needs to be retained; The target data generation module is specifically configured to: extract the source data fields that need to be retained from the source data according to the read conversion annotation information, and convert the field names into corresponding target field names to obtain target data. 8. The device according to claim 6, wherein the conversion annotation information further comprises: the target field type after the conversion of the source data field that needs to be retained; The target data generation module is specifically configured to: extract the source data fields to be retained from the source data according to the read conversion annotation information, and convert the field types to corresponding target field types to obtain target data. 9. The apparatus of claim 6, wherein the apparatus further comprises: The saving module is used for saving the target data as single-valued data, and the value of the single-valued data carries the key-value pair information of each field of the target data. 10 . The apparatus according to claim 9 , wherein the value of the single-valued data further carries field type information of each field of the target data. 11 .

Images