CN114840586A - Data conversion method, equipment, server and computer readable storage medium - Google Patents

Data conversion method, equipment, server and computer readable storage medium Download PDF

Info

Publication number
CN114840586A
CN114840586A CN202210376854.4A CN202210376854A CN114840586A CN 114840586 A CN114840586 A CN 114840586A CN 202210376854 A CN202210376854 A CN 202210376854A CN 114840586 A CN114840586 A CN 114840586A
Authority
CN
China
Prior art keywords
data
field
identifier
conversion
format
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210376854.4A
Other languages
Chinese (zh)
Inventor
赵帅
唐川
钟科
官旭
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Weikun Shanghai Technology Service Co Ltd
Original Assignee
Weikun Shanghai Technology Service Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Weikun Shanghai Technology Service Co Ltd filed Critical Weikun Shanghai Technology Service Co Ltd
Priority to CN202210376854.4A priority Critical patent/CN114840586A/en
Publication of CN114840586A publication Critical patent/CN114840586A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25—Integrating or interfacing systems involving database management systems
    • G06F16/258—Data format conversion from or to a database
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22—Indexing; Data structures therefor; Storage structures
    • G06F16/2282—Tablespace storage structures; Management thereof

Abstract

The embodiment of the invention discloses a data conversion method, equipment, a server and a computer readable storage medium, wherein the method comprises the following steps: acquiring data conversion information configured by a user on a user configuration interface, wherein the data conversion information comprises a field matching mapping relation between a first field in a first data table and a second field in a second data table, and a conversion rule between a data format of the first field and a data format of the second field; comparing the field data in the first data table with the field data in the second data table according to the field matching mapping relation, and determining first field data in the first data table, which is inconsistent with the data format of the field data in the second data table, and second field data in the second data table, which corresponds to the first field data; and converting the data format of the first field data and/or the data format of the second field data according to the conversion rule. The data conversion efficiency can be improved by the embodiment.

Description

Data conversion method, equipment, server and computer readable storage medium
Technical Field
The present invention relates to the field of data conversion technologies, and in particular, to a data conversion method, device, server, and computer-readable storage medium.
Background
The data in the data warehouse is a collection of data oriented to a certain subject, the data is extracted from a plurality of business systems and contains historical data, and therefore the condition that some data are wrong data and some data have conflicts among data formats is avoided, and the wrong or conflicting data are called 'dirty data'.
At present, the operation of cleaning and converting data is covered by the conversion operation of 'dirty data', and the operation usually requires a user to find problems, then the dirty data is inquired out, a processing function is written into a database for conversion, and finally the converted data is written into a target table. The user has to repeatedly write the conversion operation function and set the input and output relationship in the whole process, which is not only inefficient, but also easy to cause errors. Therefore, how to more efficiently implement data conversion becomes an urgent problem to be solved.
Disclosure of Invention
The embodiment of the invention provides a data conversion method, data conversion equipment, a server and a computer readable storage medium, which can improve the efficiency of data conversion.
In a first aspect, an embodiment of the present invention provides a data conversion method, including:
receiving a data processing request sent by a terminal, wherein the data processing request carries a first data table identifier, and determining a first data table corresponding to the first data table identifier according to the first data table identifier;
determining a second data table identifier corresponding to the first data table identifier according to a preset mapping relation of the data table identifiers, and determining a second data table corresponding to the second data table identifier;
acquiring data conversion information configured by a user on a user configuration interface, wherein the data conversion information comprises a field matching mapping relation between a first field in a first data table and a second field in a second data table, and a conversion rule between a data format of the first field and a data format of the second field;
comparing the field data in the first data table with the field data in the second data table according to the field matching mapping relation, and determining first field data in the first data table, which is inconsistent with the data format of the field data in the second data table, and second field data in the second data table, which corresponds to the first field data;
And converting the data format of the first field data and/or the data format of the second field data according to the conversion rule.
Further, the conversion rule carries a target data format; the converting the data format of the first field data and/or the data format of the second field data according to the conversion rule includes:
determining whether the target data format matches a data format of the first field data;
if the target data format is matched with the data format of the first field data, determining a first conversion operation function according to the conversion rule and the data format of the first field data;
and converting the data format of the second field data into the data format of the first field data according to the first conversion operation function.
Further, the converting the data format of the first field data and/or the data format of the second field data according to the conversion rule includes:
if the target data format does not match the data format of the first field data, determining whether the target data format matches the data format of the second field data;
When the target data format is matched with the data format of the second field data, determining a second conversion operation function according to the conversion rule and the data format of the second field data;
and converting the data format of the first field data into the data format of the second field data according to the second conversion operation function.
Further, the converting the data format of the first field data and/or the data format of the second field data according to the conversion rule includes:
if the target data format is determined not to be matched with the data format of the first field data and the data format of the second field data, determining a third conversion operation function according to the conversion rule and the target data format;
and converting the data format of the first field data and the data format of the second field data into the target data format according to the third conversion operation function.
Further, before the obtaining of the data conversion information configured by the user at the user configuration interface, the method further includes:
outputting a first user configuration interface, wherein the first user configuration interface comprises a first parameter input area, and the first parameter input area is used for a user to input a conversion rule identifier, a conversion operation function, a parameter value and description information;
And acquiring a conversion rule identifier, a conversion operation function, a parameter value and description information input by a user on the first user configuration interface, and generating the data conversion information according to the conversion rule identifier, the conversion operation function, the parameter value and the description information.
Further, before the acquiring data conversion information configured by the user in the user configuration interface, the method further includes:
outputting a second user configuration interface, wherein the second user configuration interface comprises a second parameter input area and a third parameter input area, the second parameter input area is used for a user to input a first data table identifier and a second data table identifier, the third parameter input area is used for the user to input a first field identifier and a second field identifier, the first field identifier corresponds to the first field data, and the second field identifier corresponds to the second field data;
acquiring a first data table identifier and a second data table identifier input by a user in the second parameter input area, determining a first data table according to the first data table identifier and determining a second data table according to the second data table identifier;
and acquiring a first field identifier and a second field identifier input by a user in the third parameter input area, and generating the field matching mapping relation according to the first field identifier and the second field identifier.
Further, the second user configuration interface further includes a fourth parameter input area, where the fourth parameter input area is used for a user to select a conversion rule identifier; the method further comprises the following steps:
and acquiring a conversion rule identifier selected by a user in the fourth parameter input area, and determining a conversion rule for converting the data format of the first field data and/or the data format of the second field data according to the conversion rule identifier.
In a second aspect, an embodiment of the present invention provides a data conversion apparatus, which includes a unit configured to execute the data conversion method of the first aspect.
In a third aspect, an embodiment of the present invention provides a server, including a processor, an input device, an output device, and a memory, where the processor, the input device, the output device, and the memory are connected to each other, where the memory is used to store a computer program that supports a data conversion device to execute the method described above, and the computer program includes a program, and the processor is configured to call the program to execute the method described above in the first aspect.
In a fourth aspect, the present invention provides a computer-readable storage medium, which stores a computer program, where the computer program is executed by a processor to implement the method of the first aspect.
In the embodiment of the present invention, a data conversion device may obtain data conversion information configured by a user on a user configuration interface, where the data conversion information includes a field matching mapping relationship between a first field in a first data table and a second field in a second data table, and a conversion rule between a data format of the first field and a data format of the second field; comparing the field data in the first data table with the field data in the second data table according to the field matching mapping relation, and determining first field data in the first data table, which is inconsistent with the data format of the field data in the second data table, and second field data in the second data table, which is corresponding to the first field data; and converting the data format of the first field data and/or the data format of the second field data according to the conversion rule. By this embodiment, the efficiency of data conversion can be improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 is a schematic flow chart diagram of a data conversion method provided by an embodiment of the invention;
FIG. 2 is a schematic diagram of a user configuration interface provided by an embodiment of the invention;
FIG. 3 is a schematic diagram of another user configuration interface provided by embodiments of the present invention;
FIG. 4 is a schematic flow chart diagram of another data conversion method provided by an embodiment of the invention;
FIG. 5 is a schematic block diagram of a data conversion apparatus provided by an embodiment of the present invention;
FIG. 6 is a schematic block diagram of a server provided by an embodiment of the present invention;
FIG. 7 is a schematic diagram of yet another user configuration interface provided by an embodiment of the invention;
fig. 8 is a schematic diagram of still another user configuration interface provided by an embodiment of the invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The data conversion method provided by the embodiment of the present invention may be executed by a data conversion device, where the data conversion device may be disposed on a server, and in some embodiments, the data conversion device may be disposed at: the mobile phone, the computer, the tablet, the intelligent watch and other intelligent terminals. In some embodiments, the data conversion device may be installed on a server; in some embodiments, the data conversion device may be spatially independent of the server; in some embodiments, the data conversion device may be a component of the server, i.e. the server comprises a data conversion device. In some embodiments, the server includes a distributed file system disposed in a Hadoop architecture including a data warehouse tool Hive.
In the embodiment of the invention, data conversion equipment can receive a data processing request sent by a terminal, wherein the data processing request carries a first data table identifier, and a first data table corresponding to the first data table identifier is determined according to the first data table identifier; and determining a second data table identifier corresponding to the first data table identifier according to a preset data table identifier mapping relation, and determining a second data table corresponding to the second data table identifier. The data conversion equipment can acquire data conversion information configured by a user on a user configuration interface, wherein the data conversion information comprises a field matching mapping relation between a first field in a first data table and a second field in a second data table, and a conversion rule between a data format of the first field and a data format of the second field; comparing the field data in the first data table with the field data in the second data table according to the field matching mapping relation, and determining first field data in the first data table, which is inconsistent with the data format of the field data in the second data table, and second field data in the second data table, which corresponds to the first field data; and converting the data format of the first field data and/or the data format of the second field data according to the conversion rule.
According to the scheme, data conversion information among the field data of the data table of each Hive database is configured in advance, and the data format of the first field data or the second field data to be converted is converted according to the data conversion information, so that the data conversion efficiency is improved. The data conversion method according to the embodiment of the present invention is schematically described below with reference to the drawings.
Referring to fig. 1, fig. 1 is a schematic flow chart of a data conversion method according to an embodiment of the present invention, and as shown in fig. 1, the method may be executed by a data conversion device, and a specific explanation of the data conversion device is as described above, which is not repeated herein. Specifically, the method of the embodiment of the present invention includes the following steps.
S101: receiving a data processing request sent by a terminal, wherein the data processing request carries a first data table identifier, and determining a first data table corresponding to the first data table identifier according to the first data table identifier.
In the embodiment of the present invention, a data processing device may receive a data processing request sent by a terminal, where the data processing request carries a first data table identifier, and determine, according to the first data table identifier, a first data table corresponding to the first data table identifier, where the first data table includes multiple first field data.
S102: and determining a second data table identifier corresponding to the first data table identifier according to a preset data table identifier mapping relation, and determining a second data table corresponding to the second data table identifier.
In this embodiment of the present invention, the data processing device may determine, according to a preset mapping relationship of data table identifiers, a second data table identifier corresponding to the first data table identifier, and determine a second data table corresponding to the second data table identifier. In some embodiments, in the preset mapping relationship of the data table identifiers, the first data table identifier may be the same as the second data table identifier; in some embodiments, in the preset mapping relationship of the data table identifiers, the first data table identifier may correspond to a different data table identifier than the second data table identifier. In some embodiments, the data table identification may include, but is not limited to, any one or more characters of numbers, letters, words, and the like.
In some embodiments, the first data table and the second data table may be different data tables in the same database; in some embodiments, the first data table and the second data table may be different data tables in different databases.
S103: the method comprises the steps of obtaining data conversion information configured by a user on a user configuration interface, wherein the data conversion information comprises a field matching mapping relation between a first field in a first data table and a second field in a second data table, and a conversion rule between a data format of the first field and a data format of the second field.
In some embodiments, the data conversion information includes a field matching mapping relationship between a first field in a first data table and a second field in a second data table, and a conversion rule between a data format of the first field and a data format of the second field.
In one embodiment, the data conversion device may output a user configuration interface before acquiring data conversion information configured by a user on the user configuration interface, where the user configuration interface is used by the user to configure the data conversion information, and acquires the data conversion information configured by the user on the user configuration interface, where the data conversion information includes a field matching mapping relationship between a first field in a first data table and a second field in a second data table, and a conversion rule between a data format of the first field and a data format of the second field. In some embodiments, the user configuration interface may include a plurality of interfaces, and the embodiments of the present invention are not limited in particular.
S104: and comparing the field data in the first data table with the field data in the second data table according to the field matching mapping relation, and determining first field data in the first data table, which is inconsistent with the data format of the field data in the second data table, and second field data in the second data table, which is corresponding to the first field data.
In this embodiment of the present invention, the data conversion device may compare the field data in the first data table with the field data in the second data table according to the field matching mapping relationship, and determine first field data in the first data table that is inconsistent with the data format of the field data in the second data table and second field data in the second data table that corresponds to the first field data.
In one embodiment, the data conversion device may compare the field data in the first data table with the corresponding field data in the second data table according to the field matching mapping relationship, determine whether the data formats of the field data in the first data table and the corresponding field data in the second data table are consistent, and if the data formats of the field data in the first data table and the corresponding field data in the second data table are inconsistent, obtain the first field data with inconsistent data formats in the first data table and the second field data with inconsistent data formats in the second data table.
S105: and converting the data format of the first field data and/or the data format of the second field data according to the conversion rule.
In this embodiment of the present invention, the data conversion device may convert the data format of the first field data and/or the data format of the second field data according to the conversion rule. In some embodiments, the conversion rule may be entered by a user on a user configuration interface; in some embodiments, the conversion rule may also be preset, and is not specifically limited herein.
In one embodiment, the conversion rules carry a target data format; the data conversion device may determine whether the target data format matches the data format of the first field data when converting the data format of the first field data and/or the data format of the second field data according to the conversion rule; if the target data format is matched with the data format of the first field data, determining a first conversion operation function according to the conversion rule and the data format of the first field data; and converting the data format of the second field data into the data format of the first field data according to the first conversion operation function.
In one embodiment, when the data conversion device converts the data format of the first field data and/or the data format of the second field data according to the conversion rule, if the target data format does not match the data format of the first field data, it may further determine whether the target data format matches the data format of the second field data; when the target data format is matched with the data format of the second field data, determining a second conversion operation function according to the conversion rule and the data format of the second field data; and converting the data format of the first field data into the data format of the second field data according to the second conversion operation function.
In one embodiment, when the data conversion device converts the data format of the first field data and/or the data format of the second field data according to the conversion rule, if it is determined that the target data format does not match the data format of the first field data or the data format of the second field data, a third conversion operation function may be determined according to the conversion rule and the target data format; and converting the data format of the first field data and the data format of the second field data into the target data format according to the third conversion operation function.
In one embodiment, after converting the data format of the first field data and/or the data format of the second field data, the data conversion device may further perform a replacement process on the converted first field data and the converted second field data according to the field matching mapping relationship.
In one embodiment, after converting the data format of the first field data and/or the data format of the second field data, the data conversion device may further perform a replacement process on the converted second field data and the first field data according to the field matching mapping relationship.
In one embodiment, after converting the data format of the first field data and/or the data format of the second field data, the data conversion device may further perform a replacement process on the converted second field data and the converted first field data according to the field matching mapping relationship.
In one embodiment, the data conversion device may store the converted first field data in the first data table into the second data table according to the field matching mapping relationship, and perform the overwriting processing on the converted second field data in the second data table.
In one embodiment, the data conversion device may match each first field data in the first data table with a preset encryption field data and determine a first field data to be encrypted in the first data table that matches the preset encryption field data before performing a replacement process on the converted first field data and the second field data or performing a replacement process on the converted second field data and the first field data according to the field matching mapping relationship; and further encrypting the first field data to be encrypted according to an encryption algorithm carried in the preset encrypted field data to obtain the encrypted first field data.
In an embodiment, when encrypting the first field data to be encrypted according to an encryption algorithm carried in the preset encrypted field data, the data conversion device may determine the number of the first field data to be encrypted in the first data table, which matches the preset encrypted field data; if the number of the first field data to be encrypted is smaller than a preset threshold value, encrypting each first field data to be encrypted respectively according to an encryption algorithm carried in the preset encrypted field data; if the number of the first field data to be encrypted is greater than or equal to the preset threshold, all the first field data to be encrypted can be packaged to obtain a data packet, and the data packet is encrypted according to an encryption algorithm carried in the preset encrypted field data.
In the embodiment of the present invention, a data conversion device may obtain data conversion information configured by a user on a user configuration interface, where the data conversion information includes a field matching mapping relationship between a first field in a first data table and a second field in a second data table, and a conversion rule between a data format of the first field and a data format of the second field; comparing the field data in the first data table with the field data in the second data table according to the field matching mapping relation, and determining first field data in the first data table, which is inconsistent with the data format of the field data in the second data table, and second field data in the second data table, which corresponds to the first field data; and converting the data format of the first field data and/or the data format of the second field data according to the conversion rule. By this embodiment, the efficiency of data conversion can be improved.
Referring to fig. 4, fig. 4 is a schematic flowchart of another data conversion method according to an embodiment of the present invention, and as shown in fig. 4, the method may be executed by a data conversion device, and a specific explanation of the data conversion device is as described above, and is not described herein again. The embodiment of the present invention is different from the embodiment described in fig. 1, in that the embodiment of the present invention schematically illustrates a specific implementation process of the data conversion method. Specifically, the method of the embodiment of the present invention includes the following steps.
S401: receiving a data processing request sent by a terminal, wherein the data processing request carries a first data table identifier, and determining a first data table corresponding to the first data table identifier according to the first data table identifier.
In the embodiment of the present invention, the data conversion device may receive a data processing request sent by the terminal, where the data processing request carries a first data table identifier, and determine, according to the first data table identifier, a first data table corresponding to the first data table identifier.
S402: and determining a second data table identifier corresponding to the first data table identifier according to a preset data table identifier mapping relation, and determining a second data table corresponding to the second data table identifier.
In the embodiment of the present invention, the data conversion device may determine, according to a preset mapping relationship of the data table identifier, a second data table identifier corresponding to the first data table identifier, and determine a second data table corresponding to the second data table identifier.
S403: and outputting a user configuration interface, wherein the user configuration interface is used for configuring data conversion information by a user and acquiring the data conversion information configured by the user on the user configuration interface, and the data conversion information comprises a field matching mapping relation between a first field in a first data table and a second field in a second data table and a conversion rule between a data format of the first field and a data format of the second field.
In the embodiment of the present invention, the data conversion device may output a user configuration interface, where the user configuration interface is used by a user to configure data conversion information, and acquire the data conversion information configured by the user on the user configuration interface, where the data conversion information includes a field matching mapping relationship between a first field in a first data table and a second field in a second data table, and a conversion rule between a data format of the first field and a data format of the second field.
In one embodiment, the data conversion device may output a first user configuration interface including a first parameter input area for a user to input a conversion rule identification, a conversion operation function, a parameter value, and description information.
In one embodiment, the data conversion device may obtain a conversion rule identifier, a conversion operation function, a parameter value, and description information input by a user in the first user configuration interface, and generate the data conversion information according to the conversion rule identifier, the conversion operation function, the parameter value, and the description information.
In some embodiments, the conversion rule identification includes, but is not limited to, a conversion rule name; in some embodiments, the conversion function includes, but is not limited to, a string extraction function such as substring; in certain embodiments, the parameter values include, but are not limited to, parameter values for a date string, a time string; in some embodiments, the description information may include conversion description information, such as converting a time string in yyy-MM-dd hh: MM: ss format to a time string in yyy-MM-dd format.
Specifically, fig. 2 is an example, and fig. 2 is a schematic diagram of a user configuration interface according to an embodiment of the present invention. As shown in fig. 2, before acquiring data conversion information configured by a user in a user configuration interface, the data conversion apparatus may output a first user configuration interface 21, where the first user configuration interface 21 includes a first parameter input area, where the first parameter input area includes an area 211 for the user to input a conversion rule identifier, an area 212 for the user to input a conversion operation function, an area 213 for the user to input a parameter value, and an area 214 for the user to input description information. The data conversion device may identify the conversion rule entered by the user in the area 211 of the first user configuration interface 21, the conversion operation function entered in the area 212, the parameter value entered in the area 213, and the description information entered in the area 214, and generate the data conversion information according to the conversion rule identification, the conversion operation function, the parameter value, and the description information.
In one embodiment, the data conversion device may output a second user configuration interface, the second user configuration interface including a second parameter input area for a user to input a first data table identifier and a second data table identifier, and a third parameter input area for a user to input a first field identifier and a second field identifier, the first field identifier corresponding to the first field data, and the second field identifier corresponding to the second field data.
In one embodiment, the data conversion device may obtain a first data table identifier and a second data table identifier input by a user in the second parameter input area, determine a first data table according to the first data table identifier, and determine a second data table according to the second data table identifier. The data conversion device may obtain a first field identifier and a second field identifier, which are input by a user in the third parameter input area, and generate the field matching mapping relationship according to the first field identifier and the second field identifier.
Specifically, fig. 3 is an example, and fig. 3 is a schematic diagram of another user configuration interface provided in the embodiment of the present invention. As shown in fig. 3, before receiving a data conversion request sent by a terminal, the data conversion device may further output a second user configuration interface 31, where the second user configuration interface 31 includes a second parameter input area and a third parameter input area, and the second parameter input area includes an area 311 for a user to input a database identifier corresponding to the first data table; the second parameter entry area includes an area 312 for a user to enter a first data table identification. The second parameter input area comprises an area 313 used for a user to input a database identifier corresponding to the second data table; the second parameter input area includes an area 314 for the user to enter a second data table identification. The third parameter input area 315 is used for a user to input a first field identifier and a second field identifier, where the first field identifier corresponds to the first field data, and the second field identifier corresponds to the second field data. And generating a preset field matching mapping relation through the first field identification and the second field identification.
In one embodiment, the second user configuration interface further includes a fourth parameter input area, where the fourth parameter input area is used for a user to select a conversion rule identifier; the data conversion device may obtain a conversion rule identifier selected by a user in the fourth parameter input area, and determine a conversion rule for converting the data format of the first field data and/or the data format of the second field data according to the conversion rule identifier.
Specifically, as shown in fig. 3, for example, the data conversion apparatus may include a fourth parameter input area 316, where the fourth parameter input area 316 is used for a user to select a conversion rule identifier, and the data conversion apparatus may obtain the conversion rule identifier selected by the user in the fourth parameter input area 316, and determine, according to the selected conversion rule identifier, a conversion rule for converting the data format of the first field data and/or the data format of the second field data.
In one embodiment, the data conversion device may output a third user configuration interface including a fifth parameter input area for a user to input field data to be encrypted and an encryption algorithm. In an embodiment, the third user configuration interface further includes another parameter input area for inputting other configuration information, such as condition information for configuring the first data table, partition information for configuring the second data table, some preprocessing information for configuring the first data table, cleaning work information after replacing the configuration data, and the like, which is not limited herein.
In one embodiment, the data conversion apparatus may further output a fourth user configuration interface, the fourth user configuration interface including a sixth parameter input area for a user to input task job information. In some embodiments, the task job information includes a job name, a job type, a job description; the operation types comprise data transmission operation, data quality operation and Hive SQL. Through the implementation, the user can visually configure the task operation information.
Specifically, fig. 7 is an example, and fig. 7 is a schematic diagram of another user configuration interface provided in the embodiment of the present invention. As shown in fig. 7, the sixth parameter input area 70 includes a first area 701, a second area 702, and a third area 703, where the first area 701 is used to input a job name, the second area 702 is used to input a job type, and the third area 703 is used to input job description information.
In one embodiment, the fourth user configuration interface includes a seventh parameter input area, and the seventh parameter input area is used for a user to input the job flow information in the seventh parameter input area after configuring the task job information. In some embodiments, the workflow information includes a city of the workflow, a scheduling name, the workflow, a corn expression, a task description, and workflow parameters; in some embodiments, the corn expression includes a time expression, and the time information in the cron expression is an execution cycle, and is used for controlling a cycle execution plan; the job flow parameters comprise job names, parameter names and parameter values.
Specifically, fig. 8 is an example, and fig. 8 is a schematic diagram of another user configuration interface provided in the embodiment of the present invention. As shown in fig. 8, the seventh parameter input area 80 includes an area 801 for inputting a city of a job flow such as a celebration, an area 802 for inputting a schedule name such as schedule _ test, an area 803 for inputting a job flow, an area 804 for inputting a corn expression, an area 805 for inputting a task description, and an area 806 for inputting a job flow parameter (such as test-flow).
Therefore, by configuring the task operation information and the task operation flow, the task scheduling configuration can be realized, the periodic data conversion is facilitated, and the flexibility of the data conversion is improved.
S404: comparing the field data in the first data table with the field data in the second data table according to the field matching mapping relation, and determining first field data in the first data table, which is inconsistent with the data format of the field data in the second data table, and second field data in the second data table, which corresponds to the first field data.
In this embodiment of the present invention, the data conversion device may compare the field data in the first data table with the field data in the second data table according to the field matching mapping relationship, and determine first field data in the first data table that is inconsistent with the data format of the field data in the second data table and second field data in the second data table that corresponds to the first field data.
S405: and converting the data format of the first field data and/or the data format of the second field data according to the conversion rule.
In this embodiment of the present invention, the data conversion device may convert the data format of the first field data and/or the data format of the second field data according to the conversion rule.
In the embodiment of the invention, the data conversion equipment can output the user configuration interface and acquire the data conversion information configured on the user configuration interface by the user, wherein the data conversion information comprises the field matching mapping relation and the conversion rule. The data conversion device may compare the field data in the first data table with the field data in the second data table according to the field matching mapping relationship, and determine first field data in the first data table that is inconsistent with the data format of the field data in the second data table and second field data in the second data table that corresponds to the first field data; and converting the data format of the first field data and/or the data format of the second field data according to the conversion rule. By this embodiment, the efficiency of data conversion can be improved.
The embodiment of the invention also provides data conversion equipment, which is used for executing the unit of the method in any one of the preceding claims. Specifically, referring to fig. 5, fig. 5 is a schematic block diagram of a data conversion device according to an embodiment of the present invention. The data conversion apparatus of the present embodiment includes: a receiving unit 501, a first determining unit 502, an obtaining unit 503, a second determining unit 504, and a converting unit 505.
A receiving unit 501, configured to receive a data processing request sent by a terminal, where the data processing request carries a first data table identifier, and determine, according to the first data table identifier, a first data table corresponding to the first data table identifier;
a first determining unit 502, configured to determine, according to a preset mapping relationship of data table identifiers, a second data table identifier corresponding to the first data table identifier, and determine a second data table corresponding to the second data table identifier;
an obtaining unit 503, configured to obtain data conversion information configured by a user on a user configuration interface, where the data conversion information includes a field matching mapping relationship between a first field in a first data table and a second field in a second data table, and a conversion rule between a data format of the first field and a data format of the second field;
A second determining unit 504, configured to compare field data in the first data table with field data in the second data table according to the field matching mapping relationship, and determine first field data in the first data table that is inconsistent with the data format of the field data in the second data table and second field data in the second data table that corresponds to the first field data;
the converting unit 505 converts the data format of the first field data and/or the data format of the second field data according to the conversion rule.
Further, the conversion rule carries a target data format; when the converting unit 505 converts the data format of the first field data and/or the data format of the second field data according to the conversion rule, the converting unit is specifically configured to:
determining whether the target data format matches a data format of the first field data;
if the target data format is matched with the data format of the first field data, determining a first conversion operation function according to the conversion rule and the data format of the first field data;
And converting the data format of the second field data into the data format of the first field data according to the first conversion operation function.
Further, when the converting unit 505 converts the data format of the first field data and/or the data format of the second field data according to the conversion rule, specifically, the converting unit is configured to:
if the target data format does not match the data format of the first field data, determining whether the target data format matches the data format of the second field data;
when the target data format is matched with the data format of the second field data, determining a second conversion operation function according to the conversion rule and the data format of the second field data;
and converting the data format of the first field data into the data format of the second field data according to the second conversion operation function.
Further, when the converting unit 505 converts the data format of the first field data and/or the data format of the second field data according to the conversion rule, specifically, the converting unit is configured to:
if the target data format is determined not to be matched with the data format of the first field data and the data format of the second field data, determining a third conversion operation function according to the conversion rule and the target data format;
And converting the data format of the first field data and the data format of the second field data into the target data format according to the third conversion operation function.
Further, the obtaining unit 503 is further configured to, before obtaining the data conversion information configured by the user in the user configuration interface, further:
outputting a first user configuration interface, wherein the first user configuration interface comprises a first parameter input area, and the first parameter input area is used for a user to input a conversion rule identifier, a conversion operation function, a parameter value and description information;
and acquiring a conversion rule identifier, a conversion operation function, a parameter value and description information input by a user on the first user configuration interface, and generating the data conversion information according to the conversion rule identifier, the conversion operation function, the parameter value and the description information.
Further, the obtaining unit 503 is further configured to, before obtaining the data conversion information configured by the user in the user configuration interface, further:
outputting a second user configuration interface, wherein the second user configuration interface comprises a second parameter input area and a third parameter input area, the second parameter input area is used for a user to input a first data table identifier and a second data table identifier, the third parameter input area is used for the user to input a first field identifier and a second field identifier, the first field identifier corresponds to the first field data, and the second field identifier corresponds to the second field data;
Acquiring a first data table identifier and a second data table identifier input by a user in the second parameter input area, determining a first data table according to the first data table identifier and determining a second data table according to the second data table identifier;
and acquiring a first field identifier and a second field identifier input by a user in the third parameter input area, and generating the field matching mapping relation according to the first field identifier and the second field identifier.
Further, the second user configuration interface further includes a fourth parameter input area, where the fourth parameter input area is used for a user to select a conversion rule identifier; the obtaining unit 503 is further configured to:
and acquiring a conversion rule identifier selected by a user in the fourth parameter input area, and determining a conversion rule for converting the data format of the first field data and/or the data format of the second field data according to the conversion rule identifier.
In the embodiment of the present invention, a data conversion device may obtain data conversion information configured by a user on a user configuration interface, where the data conversion information includes a field matching mapping relationship between a first field in a first data table and a second field in a second data table, and a conversion rule between a data format of the first field and a data format of the second field; comparing the field data in the first data table with the field data in the second data table according to the field matching mapping relation, and determining first field data in the first data table, which is inconsistent with the data format of the field data in the second data table, and second field data in the second data table, which corresponds to the first field data; and converting the data format of the first field data and/or the data format of the second field data according to the conversion rule. By this embodiment, the efficiency of data conversion can be improved.
Referring to fig. 6, fig. 6 is a schematic block diagram of a server according to an embodiment of the present invention. The server in this embodiment as shown in the figure may include: one or more processors 601; one or more input devices 602, one or more output devices 603, and memory 604. The processor 601, the input device 602, the output device 603, and the memory 604 are connected by a bus 605. The memory 604 is used for storing computer programs, including programs, and the processor 601 is used for executing the programs stored by the memory 604.
Wherein, the processor 601 is configured to invoke the program to execute:
receiving a data processing request sent by a terminal, wherein the data processing request carries a first data table identifier, and determining a first data table corresponding to the first data table identifier according to the first data table identifier;
determining a second data table identifier corresponding to the first data table identifier according to a preset data table identifier mapping relation, and determining a second data table corresponding to the second data table identifier;
acquiring data conversion information configured by a user on a user configuration interface, wherein the data conversion information comprises a field matching mapping relation between a first field in a first data table and a second field in a second data table, and a conversion rule between a data format of the first field and a data format of the second field;
Comparing the field data in the first data table with the field data in the second data table according to the field matching mapping relation, and determining first field data in the first data table, which is inconsistent with the data format of the field data in the second data table, and second field data in the second data table, which corresponds to the first field data;
and converting the data format of the first field data and/or the data format of the second field data according to the conversion rule.
Further, the conversion rule carries a target data format; when the processor 601 converts the data format of the first field data and/or the data format of the second field data according to the conversion rule, the method is specifically configured to:
determining whether the target data format matches a data format of the first field data;
if the target data format is matched with the data format of the first field data, determining a first conversion operation function according to the conversion rule and the data format of the first field data;
and converting the data format of the second field data into the data format of the first field data according to the first conversion operation function.
Further, when the processor 601 converts the data format of the first field data and/or the data format of the second field data according to the conversion rule, the processor is specifically configured to:
if the target data format does not match the data format of the first field data, determining whether the target data format matches the data format of the second field data;
when the target data format is matched with the data format of the second field data, determining a second conversion operation function according to the conversion rule and the data format of the second field data;
and converting the data format of the first field data into the data format of the second field data according to the second conversion operation function.
Further, when the processor 601 converts the data format of the first field data and/or the data format of the second field data according to the conversion rule, the processor is specifically configured to:
if the target data format is determined not to be matched with the data format of the first field data and the data format of the second field data, determining a third conversion operation function according to the conversion rule and the target data format;
And converting the data format of the first field data and the data format of the second field data into the target data format according to the third conversion operation function.
Further, the processor 601, before obtaining the data conversion information configured by the user in the user configuration interface, is further configured to:
outputting a first user configuration interface, wherein the first user configuration interface comprises a first parameter input area, and the first parameter input area is used for a user to input a conversion rule identifier, a conversion operation function, a parameter value and description information;
and acquiring a conversion rule identifier, a conversion operation function, a parameter value and description information input by a user on the first user configuration interface, and generating the data conversion information according to the conversion rule identifier, the conversion operation function, the parameter value and the description information.
Further, the processor 601, before obtaining the data conversion information configured by the user in the user configuration interface, is further configured to:
outputting a second user configuration interface, wherein the second user configuration interface comprises a second parameter input area and a third parameter input area, the second parameter input area is used for a user to input a first data table identifier and a second data table identifier, the third parameter input area is used for the user to input a first field identifier and a second field identifier, the first field identifier corresponds to the first field data, and the second field identifier corresponds to the second field data;
Acquiring a first data table identifier and a second data table identifier input by a user in the second parameter input area, determining a first data table according to the first data table identifier and determining a second data table according to the second data table identifier;
and acquiring a first field identifier and a second field identifier input by a user in the third parameter input area, and generating the field matching mapping relation according to the first field identifier and the second field identifier.
Further, the second user configuration interface further includes a fourth parameter input area, where the fourth parameter input area is used for a user to select a conversion rule identifier; the processor 601 is further configured to:
and acquiring a conversion rule identifier selected by a user in the fourth parameter input area, and determining a conversion rule for converting the data format of the first field data and/or the data format of the second field data according to the conversion rule identifier.
In the embodiment of the invention, a server can acquire data conversion information configured by a user on a user configuration interface, wherein the data conversion information comprises a field matching mapping relation between a first field in a first data table and a second field in a second data table, and a conversion rule between a data format of the first field and a data format of the second field; comparing the field data in the first data table with the field data in the second data table according to the field matching mapping relation, and determining first field data in the first data table, which is inconsistent with the data format of the field data in the second data table, and second field data in the second data table, which corresponds to the first field data; and converting the data format of the first field data and/or the data format of the second field data according to the conversion rule. By this embodiment, the efficiency of data conversion can be improved.
It should be understood that, in the embodiment of the present invention, the Processor 601 may be a Central Processing Unit (CPU), and the Processor may also be other general processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable gate arrays (FPGAs) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The input device 602 may include a touch pad, microphone, etc., and the output device 603 may include a display (LCD, etc.), speakers, etc.
The memory 604 may include both read-only memory and random access memory, and provides instructions and data to the processor 601. A portion of the memory 604 may also include non-volatile random access memory. For example, the memory 604 may also store device type information.
In a specific implementation, the processor 601, the input device 602, and the output device 603 described in this embodiment of the present invention may execute the implementation manner described in the method embodiment shown in fig. 1 or fig. 4 of the data conversion method provided in this embodiment of the present invention, and may also execute the implementation manner of the data conversion device described in fig. 5 in this embodiment of the present invention, which is not described herein again.
The embodiment of the present invention further provides a computer-readable storage medium, where a computer program is stored, and when the computer program is executed by a processor, the data conversion method described in the embodiment corresponding to fig. 1 or fig. 4 is implemented, and the data conversion device according to the embodiment corresponding to fig. 5 of the present invention may also be implemented, which is not described herein again.
The computer readable storage medium may be an internal storage unit of the data conversion device described in any of the foregoing embodiments, for example, a hard disk or a memory of the data conversion device. The computer-readable storage medium may also be an external storage device of the data conversion device, such as a plug-in hard disk, a Smart Memory Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like provided on the data conversion device. Further, the computer-readable storage medium may also include both an internal storage unit and an external storage device of the data conversion apparatus. The computer-readable storage medium is used for storing the computer program and other programs and data required by the data conversion apparatus. The computer readable storage medium may also be used to temporarily store data that has been output or is to be output.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention essentially or partially contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned computer-readable storage media comprise: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The above description is only a part of the embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive various equivalent modifications or substitutions within the technical scope of the present invention, and these modifications or substitutions should be covered within the scope of the present invention.

Claims (10)

1. A method of data conversion, comprising:
receiving a data processing request sent by a terminal, wherein the data processing request carries a first data table identifier, and determining a first data table corresponding to the first data table identifier according to the first data table identifier;
determining a second data table identifier corresponding to the first data table identifier according to a preset data table identifier mapping relation, and determining a second data table corresponding to the second data table identifier;
acquiring data conversion information configured by a user on a user configuration interface, wherein the data conversion information comprises a field matching mapping relation between a first field in a first data table and a second field in a second data table, and a conversion rule between a data format of the first field and a data format of the second field;
comparing the field data in the first data table with the field data in the second data table according to the field matching mapping relation, and determining first field data in the first data table, which is inconsistent with the data format of the field data in the second data table, and second field data in the second data table, which corresponds to the first field data;
And converting the data format of the first field data and/or the data format of the second field data according to the conversion rule.
2. The method of claim 1, wherein the transformation rule carries a target data format; the converting the data format of the first field data and/or the data format of the second field data according to the conversion rule includes:
determining whether the target data format matches a data format of the first field data;
if the target data format is matched with the data format of the first field data, determining a first conversion operation function according to the conversion rule and the data format of the first field data;
and converting the data format of the second field data into the data format of the first field data according to the first conversion operation function.
3. The method according to claim 2, wherein the converting the data format of the first field data and/or the data format of the second field data according to the conversion rule comprises:
if the target data format does not match the data format of the first field data, determining whether the target data format matches the data format of the second field data;
When the target data format is matched with the data format of the second field data, determining a second conversion operation function according to the conversion rule and the data format of the second field data;
and converting the data format of the first field data into the data format of the second field data according to the second conversion operation function.
4. The method according to claim 3, wherein the converting the data format of the first field data and/or the data format of the second field data according to the conversion rule comprises:
if the target data format is determined not to be matched with the data format of the first field data and the data format of the second field data, determining a third conversion operation function according to the conversion rule and the target data format;
and converting the data format of the first field data and the data format of the second field data into the target data format according to the third conversion operation function.
5. The method of claim 1, wherein the obtaining data conversion information configured by a user in a user configuration interface further comprises:
Outputting a first user configuration interface, wherein the first user configuration interface comprises a first parameter input area, and the first parameter input area is used for a user to input a conversion rule identifier, a conversion operation function, a parameter value and description information;
and acquiring a conversion rule identifier, a conversion operation function, a parameter value and description information input by a user on the first user configuration interface, and generating the data conversion information according to the conversion rule identifier, the conversion operation function, the parameter value and the description information.
6. The method of claim 1, wherein the obtaining data conversion information configured by a user in a user configuration interface further comprises:
outputting a second user configuration interface, wherein the second user configuration interface comprises a second parameter input area and a third parameter input area, the second parameter input area is used for a user to input a first data table identifier and a second data table identifier, the third parameter input area is used for the user to input a first field identifier and a second field identifier, the first field identifier corresponds to the first field data, and the second field identifier corresponds to the second field data;
Acquiring a first data table identifier and a second data table identifier input by a user in the second parameter input area, determining a first data table according to the first data table identifier and determining a second data table according to the second data table identifier;
and acquiring a first field identifier and a second field identifier input by a user in the third parameter input area, and generating the field matching mapping relation according to the first field identifier and the second field identifier.
7. The method of claim 6,
the second user configuration interface further comprises a fourth parameter input area, and the fourth parameter input area is used for a user to select a conversion rule identifier; the method further comprises the following steps:
and acquiring a conversion rule identifier selected by a user in the fourth parameter input area, and determining a conversion rule for converting the data format of the first field data and/or the data format of the second field data according to the conversion rule identifier.
8. A data conversion device, characterized in that it comprises means for performing the method of any of claims 1-7.
9. A server comprising a processor, an input device, an output device and a memory, the processor, the input device, the output device and the memory being interconnected, wherein the memory is configured to store a computer program, the computer program comprising a program, the processor being configured to invoke the program to perform the method according to any one of claims 1-7.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program which is executed by a processor to implement the method of any one of claims 1-7.
CN202210376854.4A 2022-04-02 2022-04-02 Data conversion method, equipment, server and computer readable storage medium Pending CN114840586A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210376854.4A CN114840586A (en) 2022-04-02 2022-04-02 Data conversion method, equipment, server and computer readable storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210376854.4A CN114840586A (en) 2022-04-02 2022-04-02 Data conversion method, equipment, server and computer readable storage medium

Publications (1)

Publication Number Publication Date
CN114840586A true CN114840586A (en) 2022-08-02

Family

ID=82563124

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210376854.4A Pending CN114840586A (en) 2022-04-02 2022-04-02 Data conversion method, equipment, server and computer readable storage medium

Country Status (1)

Country Link
CN (1) CN114840586A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116483475A (en) * 2023-06-26 2023-07-25 北京营安科技有限公司 Configuration method and device of data conversion information, electronic equipment and storage medium

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116483475A (en) * 2023-06-26 2023-07-25 北京营安科技有限公司 Configuration method and device of data conversion information, electronic equipment and storage medium

Similar Documents

Publication Publication Date Title
CN111563368A (en) Report generation method and device, computer equipment and storage medium
CN110069572A (en) HIVE method for scheduling task, device, equipment and storage medium based on big data platform
CN111639066A (en) Data cleaning method and device
CN111104421A (en) Data query method and device based on data interface standard configuration
CN110737594B (en) Database standard conformance testing method and device for automatically generating test cases
CN110503408B (en) Business process management system based on custom configuration
CN117238433B (en) Method for automatically isolating document data based on Libreoffice
CN111784318A (en) Data processing method and device, electronic equipment and storage medium
CN110275703B (en) Method and device for assigning key value to data, computer equipment and storage medium
CN114840586A (en) Data conversion method, equipment, server and computer readable storage medium
CN110472109B (en) Dynamic data quality analysis method and platform system
CN114791927A (en) Data analysis method and device
CN105009006B (en) Programmable Logic Controller, programable controller system and execution error message generation method
CN115544050A (en) Operation log recording method, device, equipment and storage medium
CN115495527A (en) Data synchronization management system and method
WO2019062087A1 (en) Attendance check data testing method, terminal and device, and computer readable storage medium
CN111813769B (en) Data processing method and device
CN112487771B (en) Report generation method, report generation device and terminal
US20120066207A1 (en) Data combination system and data combination method
CN112579571A (en) Monitoring data configuration method, data monitoring method, device, equipment and storage medium
CN110851446A (en) Data table generation method and device, computer equipment and storage medium
CN112445811A (en) Data service method, device, storage medium and component based on SQL configuration
CN112308440A (en) Work order processing method and device, computer equipment and computer readable storage medium
CN114818645B (en) Automatic report generation method, device, equipment and medium based on data body
CN114201772B (en) Stock data desensitization method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination