CN116821217A - Data distribution conversion method, device, equipment and storage medium - Google Patents
Data distribution conversion method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN116821217A CN116821217A CN202310629208.9A CN202310629208A CN116821217A CN 116821217 A CN116821217 A CN 116821217A CN 202310629208 A CN202310629208 A CN 202310629208A CN 116821217 A CN116821217 A CN 116821217A
- Authority
- CN
- China
- Prior art keywords
- data
- configuration
- information
- conversion
- task
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000006243 chemical reaction Methods 0.000 title claims abstract description 203
- 238000000034 method Methods 0.000 title claims abstract description 50
- 238000012986 modification Methods 0.000 claims abstract description 26
- 230000004048 modification Effects 0.000 claims abstract description 26
- 230000006870 function Effects 0.000 claims description 38
- 238000004590 computer program Methods 0.000 claims description 16
- 238000004364 calculation method Methods 0.000 claims description 15
- 238000012546 transfer Methods 0.000 claims description 9
- 230000009466 transformation Effects 0.000 claims description 7
- 230000000007 visual effect Effects 0.000 abstract description 11
- 238000012545 processing Methods 0.000 description 11
- 238000004891 communication Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 3
- 238000013500 data storage Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 238000013075 data extraction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the technical field of big data and discloses a data distribution conversion method, a device, equipment and a storage medium. The method comprises the following steps: acquiring configuration task parameters according to configuration information input operation aiming at a preset configuration page; acquiring data source information according to configuration task parameters, and displaying a data table corresponding to the data source information in a preset configuration page; acquiring data conversion configuration information according to modification operation on a data table, and acquiring a configuration file in a preset data format according to configuration task parameters and the data conversion configuration information; and generating a distribution conversion task according to the configuration file, and executing the distribution conversion task to realize distribution conversion of the data. According to the technical scheme, visual display is performed on the data table, and the configuration file is generated according to the modification operation of the user, so that visual data distribution aiming at different types of databases can be realized, and the distribution conversion efficiency of the data can be improved.
Description
Technical Field
The present invention relates to the field of big data technologies, and in particular, to a method, an apparatus, a device, and a storage medium for distributing and converting data.
Background
With the gradual diversification of data use scenes by enterprises, the requirements on synchronization and distribution of data are also continuously improved. How to realize efficient and accurate data conversion and distribution is becoming one of the key research directions in the field of big data.
Currently, existing data distribution conversion methods generally utilize existing data synchronization tools, such as Sqoop and DataX, to implement data distribution from a source database to a target database; however, for the Sqoop, only data synchronization of a relational database is supported, certain limitation exists on data files, data cleaning cannot be performed, and meanwhile, the problem that expansion capacity is poor and secondary development is not facilitated exists; for DataX, because it is based on memory, there may be a limit to the amount of data, and distributed cluster deployment is not possible, and the data extraction capability is greatly affected by the machine hardware performance.
Disclosure of Invention
The invention provides a data distribution conversion method, a device, equipment and a storage medium, which can realize visual data distribution aiming at different types of databases and can improve the distribution conversion efficiency of the data.
According to an aspect of the present invention, there is provided a distribution conversion method of data, including:
Acquiring configuration task parameters according to configuration information input operation aiming at a preset configuration page;
acquiring data source information according to the configuration task parameters, and displaying a data table corresponding to the data source information in the preset configuration page;
acquiring data conversion configuration information according to the modification operation on the data table, and acquiring a configuration file in a preset data format according to the configuration task parameters and the data conversion configuration information;
and generating a distribution conversion task according to the configuration file, and executing the distribution conversion task to realize distribution conversion of data.
Optionally, the configuration task parameter includes operation parameter information and target database information, and the obtaining a configuration file in a preset data format according to the configuration task parameter and the data conversion configuration information includes:
and splicing the operation parameter information, the data source information, the data conversion configuration information and the target database information to generate a configuration file with a preset data format.
By adopting the technical scheme, the configuration file containing complete data distribution and conversion information can be obtained, so that automatic data conversion and distribution from a source database to a target database can be realized, and the efficiency of data conversion and distribution can be improved.
Optionally, generating a distribution conversion task according to the configuration file includes:
according to the operation parameter information, carrying out operation parameter configuration on a preset calculation engine to obtain a target calculation engine;
and generating a distribution conversion task according to the data source information, the data conversion configuration information and the target database information through the target calculation engine.
By adopting the technical scheme, the automatic distribution of the data set with large data volume can be realized by utilizing the powerful computing capability of the computing engine.
Optionally, the distributing conversion task is executed to realize distributing conversion of the data, which includes:
and applying for obtaining a plurality of task execution nodes through a preset task scheduling frame, and executing the distribution conversion task through each task execution node so as to realize the distribution conversion of data.
By adopting the technical scheme, distributed data conversion and distribution can be realized, the data processing pressure of a single node can be reduced, and the limitation of the data storage space of the single node can be overcome.
Optionally, the distributing conversion task is executed through each task execution node, so as to implement distributing conversion of data, including:
Acquiring a data table corresponding to the data source information according to the distribution conversion task through each task execution node;
converting the data table according to the data conversion configuration information to obtain a conversion data table;
and storing the conversion data table into a target database corresponding to the target database information.
By adopting the technical scheme, the real-time synchronization of the adding and deleting operations aiming at the source database can be realized, and the data conversion efficiency can be improved.
Optionally, the data conversion configuration information includes read data information including an original column name, an original column type, an original column length, and/or a default value, and write data information including a modified column name, a modified column type, a modified column length, and/or a data conversion function;
wherein the data transfer function includes at least one of a field cut function, a field aggregate function, a field statistics function, and a date transfer function.
By adopting the technical scheme, the data conversion efficiency can be further improved, and the time spent on data conversion is shortened.
Optionally, the operation parameter information includes at least one of an operation mode, an application name, an actuator core number and a memory; the data source information comprises a data source type and data source connection information, and the data source connection information comprises at least one of a user name, a password, an access path, a database identifier and a data table identifier; the target database information includes at least one of a target library type, target library connection information, and a write mode.
By adopting the technical scheme, the online setting of data distribution can be realized, and the efficiency of data distribution can be further improved.
According to another aspect of the present invention, there is provided a distribution conversion apparatus of data, including:
the configuration task parameter acquisition module is used for acquiring configuration task parameters according to configuration information input operation aiming at a preset configuration page;
the data table display module is used for acquiring data source information according to the configuration task parameters and displaying a data table corresponding to the data source information in the preset configuration page;
the configuration file acquisition module is used for acquiring data conversion configuration information according to the modification operation on the data table, and acquiring a configuration file in a preset data format according to the configuration task parameters and the data conversion configuration information;
and the distribution conversion task generation module is used for generating a distribution conversion task according to the configuration file and realizing distribution conversion of data by executing the distribution conversion task.
According to another aspect of the present invention, there is provided an electronic apparatus including:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,,
The memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the data distribution conversion method according to any one of the embodiments of the present invention.
According to another aspect of the present invention, there is provided a computer readable storage medium storing computer instructions for causing a processor to implement a data distribution conversion method according to any one of the embodiments of the present invention when executed.
According to the technical scheme, the configuration task parameters are obtained through input operation according to configuration information aiming at the preset configuration page; then, according to the configuration task parameters, acquiring data source information, and displaying a data table corresponding to the data source information in a preset configuration page; further, according to the modification operation for the data table, acquiring data conversion configuration information, and according to the configuration task parameters and the data conversion configuration information, acquiring a configuration file in a preset data format; finally, a distribution conversion task is generated according to the configuration file, the distribution conversion task is executed to realize the distribution conversion of the data, the visual display is carried out on the data table, the configuration file is generated according to the modification operation of the user, the visual data distribution aiming at different types of databases can be realized, and the distribution conversion efficiency of the data can be improved.
It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the invention or to delineate the scope of the invention. Other features of the present invention will become apparent from the description that follows.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a flowchart of a data distribution conversion method according to a first embodiment of the present invention;
fig. 2A is a flowchart of a method for distributing and converting data according to a second embodiment of the present invention;
fig. 2B is a flow chart of another method for distributing and converting data according to the second embodiment of the present invention;
fig. 3 is a schematic structural diagram of a data distribution conversion device according to a third embodiment of the present invention;
fig. 4 is a schematic structural diagram of an electronic device implementing a data distribution conversion method according to an embodiment of the present invention.
Detailed Description
In order that those skilled in the art will better understand the present invention, a technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.
It should be noted that the terms "first," "second," "target," and the like in the description and claims of the present invention and in the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Example 1
Fig. 1 is a flowchart of a data distribution and conversion method according to an embodiment of the present invention, where the method may be applicable to a case of data distribution and conversion between different databases, and the method may be performed by a data distribution and conversion device, which may be implemented in hardware and/or software, where the data distribution and conversion device may be configured in an electronic device, typically, a computer device or a server. As shown in fig. 1, the method includes:
s110, acquiring configuration task parameters according to configuration information input operation aiming at a preset configuration page.
The preset configuration page may be a preset UI (User Interface) Interface of the distribution system; in this embodiment, when the user needs to perform the data distribution operation, the corresponding configuration information, for example, the data source type, the data source information, the target database information, and the like, may be selected or input in the preset configuration page. Thus, the distribution system can acquire the configuration task parameters set by the user.
The configuration task parameters may include, among other things, data source information, i.e., source information of the data to be distributed, such as database identification, access address, data table identification, etc. Secondly, the configuration task parameters can also comprise operation parameter information and target database information; the operation parameter information may be preset operation parameters of the big data calculation engine, for example, an operation mode, the number of executor cores, and the like; the target database information may be information of a database to which data is distributed, for example, connection information, a writing method, or the like.
S120, acquiring data source information according to the configuration task parameters, and displaying a data table corresponding to the data source information in the preset configuration page.
The data source information may include connection information of the data source, a storage path and an identifier of the data, and the like. The data sources may be file systems or different types of databases, e.g., relational and non-relational, etc. The data stored in the data source can be business data, internet of things data or government data and the like.
In this embodiment, the distribution system may parse the configuration task parameters to obtain data source information, and may determine, according to the data source information, a data source that needs to perform data distribution and a corresponding data table to be distributed. The data table may then be extracted from the data source and visually presented in the preset configuration page.
Alternatively, when the data table is visually displayed, only column attribute information of the data table, for example, column name, column type, column length, and the like may be displayed. Alternatively, only a part of the data contents (field mapping information) of the data table may be presented, and a button for viewing the entire data table may be provided; if the user wants to view all of the data sheets, he can click on the select button. In the present embodiment, the presentation form of the data table is not particularly limited.
S130, acquiring data conversion configuration information according to modification operation on the data table, and acquiring a configuration file in a preset data format according to the configuration task parameters and the data conversion configuration information.
In this embodiment, after the data table is displayed on the preset configuration page, the user may modify the displayed data table, for example, modify the column name, column type, column length, and numerical value; the distribution system can acquire corresponding data conversion configuration information according to the modification operation of the user.
Wherein the data conversion configuration information may include read data information and write data information, the read data information may include an original column name, an original column type, an original column length, and/or a default value, and the write data information may include a modified column name, a modified column type, a modified column length, and/or a data conversion function; wherein the data transfer function may include at least one of a field cut function, a field aggregate function, a field statistics function, and a date transfer function.
In this embodiment, the read data information may be attribute information of the original data table without any modification; correspondingly, the written data information may be attribute information of the modified data table. Second, the write data information may also include the data transfer function used in the modification process. The data conversion function may be pre-packaged code for implementing a specific data processing function.
In a specific example, a series of data conversion functions may be preconfigured, and when the user modifies the data table, the corresponding data conversion function may be selected in a preset configuration page according to the need, so as to implement the modification operation on the data table. Then, the distribution system can extract the numerical value corresponding to each data item in the data conversion configuration information according to the data table before and after the user modifies and the data conversion function selected in the modification process, so as to generate the final data conversion configuration information.
Secondly, after the data conversion configuration information is obtained, the data conversion configuration information and the configuration task parameters can be directly spliced to generate a configuration file with a preset data format; or the data conversion configuration information and the configuration task parameters can be subjected to format conversion and then spliced to generate the configuration file. The client may then send the configuration file to the corresponding server. The preset data format may be a pre-specified lightweight data exchange format, for example, json format, etc. The configuration file may be a parameter setting file of a distribution conversion task of data.
And S140, generating a distribution conversion task according to the configuration file, and executing the distribution conversion task to realize distribution conversion of the data.
In this embodiment, task parameter setting may be performed based on various pieces of information in the configuration file to generate a distribution conversion task; the distribution transformation task may then be scheduled to be automatically performed to extract the data to be distributed from the data source and store the data in the corresponding target library after modification.
In a specific example, the server may generate, through a preset big data computing engine Spark, a corresponding Spark task as a distribution conversion task based on data source information, operation parameter information, target database information and data conversion configuration information in the configuration file. Further, the distributed execution of the distribution conversion task by a plurality of execution nodes can be automatically scheduled to realize data distribution conversion.
When the distribution conversion task is executed, the operation parameters of Spark can be firstly configured based on the operation parameter information, then the data table corresponding to the data source information can be read from the data source, the data table is modified based on the data conversion configuration information, and finally the modified data table can be stored in the target database corresponding to the target database information.
According to the technical scheme, the configuration task parameters are obtained through input operation according to configuration information aiming at the preset configuration page; then, according to the configuration task parameters, acquiring data source information, and displaying a data table corresponding to the data source information in a preset configuration page; further, according to the modification operation for the data table, acquiring data conversion configuration information, and according to the configuration task parameters and the data conversion configuration information, acquiring a configuration file in a preset data format; finally, a distribution conversion task is generated according to the configuration file, the distribution conversion task is executed to realize the distribution conversion of the data, the visual display is carried out on the data table, the configuration file is generated according to the modification operation of the user, the visual data distribution aiming at different types of databases can be realized, and the distribution conversion efficiency of the data can be improved.
In an optional implementation manner of this embodiment, according to the configuration task parameter and the data conversion configuration information, obtaining a configuration file in a preset data format may include:
and splicing the operation parameter information, the data source information, the data conversion configuration information and the target database information to generate a configuration file with a preset data format.
In a specific example, the operation parameter information, the data source information, the data conversion configuration information and the target database information may be directly spliced according to a preset sequence to generate a configuration file in Json format.
Wherein the operating parameter information may include at least one of an operating mode (e.g., a client mode, a cluster mode, etc.), an application name, an actuator core number, and a memory; the data source information may include a data source type and data source connection information, and the data source connection information may include at least one of a user name, a password, an access path, a database identifier, and a data table identifier; the target database information may include at least one of a target library type, target library connection information, and a write mode. The operating parameter information may be an operating parameter of a pre-deployed big data compute engine.
It will be appreciated that the different types of target databases support different operations and configurations are also subject to variation. The writing mode may include incremental writing or full writing.
It should be noted that the data source type may include a file system, a relational database, a non-relational database, and the like; typically, when the data source is a MySQL database, the corresponding data source connection information may include a user name, a password, a uniform resource locator path, a database, a table name, and the like. In this embodiment, based on the data source information, the data source and corresponding specific data may be determined.
In another optional implementation manner of this embodiment, generating the distribution conversion task according to the configuration file may include:
according to the operation parameter information, carrying out operation parameter configuration on a preset calculation engine to obtain a target calculation engine;
and generating a distribution conversion task according to the data source information, the data conversion configuration information and the target database information through the target calculation engine.
The preset calculation engine may be a preset big data calculation engine corresponding to the initial parameter, for example Spark, etc. In a specific example, after obtaining the configuration file, the server may read and parse the configuration file to obtain the operating parameter information, the data source information, the data conversion configuration information, and the target database information; the initial parameters of the preset compute engine may then be modified to the operating parameter information to obtain a configured target compute engine. Further, the target calculation engine can register the data conversion function transfrom in the data conversion configuration information as a custom function, and splice the data source information, the custom function and the target database information into an SQL instruction to serve as a distribution conversion task.
Example two
Fig. 2A is a flowchart of a data distribution conversion method provided in a second embodiment of the present invention, where the technical solution in this embodiment may be combined with one or more of the foregoing embodiments. As shown in fig. 2A, the method includes:
s210, acquiring configuration task parameters according to configuration information input operation aiming at a preset configuration page.
S220, acquiring data source information according to the configuration task parameters, and displaying a data table corresponding to the data source information in the preset configuration page.
S230, acquiring data conversion configuration information according to modification operation on the data table, and splicing the operation parameter information, the data source information, the data conversion configuration information and the target database information to generate a configuration file in a preset data format.
S240, generating a distribution conversion task according to the configuration file.
S250, applying for obtaining a plurality of task execution nodes through a preset task scheduling frame, and executing the distribution conversion task through each task execution node so as to realize distribution conversion of data.
The preset task scheduling frame may be a preset functional module for task scheduling execution, for example, may be yarn or the like. In one specific example, resources may be applied to the yarn platform for the distribution transformation task; yarn may assign a plurality of task execution nodes to the distribution conversion task after detecting that the application is validated. Then, the distribution conversion task can be distributed by each task execution node so as to realize the distribution conversion of the data. The task execution node may be a server or a virtual machine.
Optionally, before data distribution, the data table to be distributed may be cleaned to wash out nonstandard data, so as to obtain a preprocessed data table.
According to the technical scheme, after distribution conversion tasks are generated according to the configuration files, a plurality of task execution nodes are obtained through application of a preset task scheduling frame, and the distribution conversion tasks are executed through the task execution nodes, so that distribution conversion of data is achieved; by adopting a plurality of distributed task execution nodes to execute distribution conversion tasks, distributed data conversion and distribution can be realized, the data processing pressure of a single node can be reduced, the limitation of the data storage space of the single node can be overcome, and the efficiency of data conversion and distribution can be further improved.
In an optional implementation manner of this embodiment, executing, by each of the task execution nodes, the distribution conversion task to implement distribution conversion of data may include:
acquiring a data table corresponding to the data source information according to the distribution conversion task through each task execution node;
converting the data table according to the data conversion configuration information to obtain a conversion data table;
And storing the conversion data table into a target database corresponding to the target database information.
In a specific example, when distributed execution of a distribution conversion task is performed by a plurality of task execution nodes, a data table corresponding to the data source information may be first extracted from a data source; then, according to the data conversion function in the data conversion configuration information, a corresponding custom function is locally registered, and based on the custom function, a data modification operation corresponding to the data conversion configuration information is executed on the acquired data table, so as to acquire a conversion data table; and finally, analyzing the target database information to acquire a target database and a corresponding writing mode, and storing the conversion data table into the target database by adopting the writing mode.
In a specific implementation manner of this embodiment, the distribution conversion flow of data may be as shown in fig. 2B; specifically, firstly, a user can configure distribution task parameters in a UI page, in this process, a server can send a data table corresponding to data source information to the UI page for display, and can record modification operation of the user on the data table, and generate corresponding data conversion configuration information.
Then, the background service may splice the operation parameter information, the data source information, the data conversion configuration information, and the target database information to generate a Json format configuration file, and start the distribution service. Further, after the distribution service is successfully started, various information in the configuration file can be analyzed and acquired to generate a spark task, and the spark task is executed through a plurality of task execution nodes so as to realize distribution conversion of data.
For example, an SQL instruction may be generated based on the data source information, the data conversion configuration information, and the target database information to extract a data table corresponding to the data source information from the data source, and perform a data modification operation corresponding to the data conversion configuration information on the data table to obtain a converted data table; finally, the conversion data table may be stored in the target database in a writing manner suitable for the target database.
The device has the advantages that visual operation data can be read, converted and written, the complexity of data processing work can be reduced, and meanwhile, the processing efficiency of a data set with a large data volume can be improved by relying on the calculation capability of spark.
Example III
Fig. 3 is a schematic structural diagram of a data distribution conversion device according to a third embodiment of the present invention. As shown in fig. 3, the apparatus includes: a configuration task parameter acquisition module 310, a data table display module 320, a configuration file acquisition module 330 and a distribution conversion task generation module 340; wherein,,
a configuration task parameter obtaining module 310, configured to obtain configuration task parameters according to a configuration information input operation for a preset configuration page;
the data table display module 320 is configured to obtain data source information according to the configuration task parameter, and display a data table corresponding to the data source information in the preset configuration page;
a configuration file obtaining module 330, configured to obtain data conversion configuration information according to a modification operation for the data table, and obtain a configuration file in a preset data format according to the configuration task parameter and the data conversion configuration information;
and the distribution conversion task generating module 340 is configured to generate a distribution conversion task according to the configuration file, and implement distribution conversion of data by executing the distribution conversion task.
According to the technical scheme, the configuration task parameters are obtained through input operation according to configuration information aiming at the preset configuration page; then, according to the configuration task parameters, acquiring data source information, and displaying a data table corresponding to the data source information in a preset configuration page; further, according to the modification operation for the data table, acquiring data conversion configuration information, and according to the configuration task parameters and the data conversion configuration information, acquiring a configuration file in a preset data format; finally, a distribution conversion task is generated according to the configuration file, the distribution conversion task is executed to realize the distribution conversion of the data, the visual display is carried out on the data table, the configuration file is generated according to the modification operation of the user, the visual data distribution aiming at different types of databases can be realized, and the distribution conversion efficiency of the data can be improved.
Optionally, the configuration task parameter includes operation parameter information and target database information, and the configuration file obtaining module 330 is specifically configured to splice the operation parameter information, the data source information, the data conversion configuration information and the target database information to generate a configuration file in a preset data format.
Optionally, the distribution conversion task generating module 340 includes:
the target computing engine acquisition unit is used for configuring the operation parameters of the preset computing engine according to the operation parameter information so as to acquire the target computing engine;
and the distribution conversion task generating unit is used for generating a distribution conversion task according to the data source information, the data conversion configuration information and the target database information through the target calculation engine.
Optionally, the distribution conversion task generating module 340 is specifically configured to apply for obtaining a plurality of task execution nodes through a preset task scheduling framework, and execute the distribution conversion task through each task execution node, so as to implement distribution conversion of data.
Optionally, the distribution conversion task generating module 340 is specifically configured to obtain, by using each task execution node, a data table corresponding to the data source information according to the distribution conversion task;
Converting the data table according to the data conversion configuration information to obtain a conversion data table;
and storing the conversion data table into a target database corresponding to the target database information.
Optionally, the data conversion configuration information includes read data information including an original column name, an original column type, an original column length, and/or a default value, and write data information including a modified column name, a modified column type, a modified column length, and/or a data conversion function;
wherein the data transfer function includes at least one of a field cut function, a field aggregate function, a field statistics function, and a date transfer function.
Optionally, the operation parameter information includes at least one of an operation mode, an application name, an actuator core number and a memory; the data source information comprises a data source type and data source connection information, and the data source connection information comprises at least one of a user name, a password, an access path, a database identifier and a data table identifier; the target database information includes at least one of a target library type, target library connection information, and a write mode.
The data distribution conversion device provided by the embodiment of the invention can execute the data distribution conversion method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.
Example IV
Fig. 4 shows a schematic diagram of an electronic device 40 that may be used to implement an embodiment of the invention. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Electronic equipment may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.
As shown in fig. 4, the electronic device 40 includes at least one processor 41, and a memory communicatively connected to the at least one processor 41, such as a Read Only Memory (ROM) 42, a Random Access Memory (RAM) 43, etc., in which the memory stores a computer program executable by the at least one processor, and the processor 41 may perform various suitable actions and processes according to the computer program stored in the Read Only Memory (ROM) 42 or the computer program loaded from the storage unit 48 into the Random Access Memory (RAM) 43. In the RAM 43, various programs and data required for the operation of the electronic device 40 may also be stored. The processor 41, the ROM 42 and the RAM 43 are connected to each other via a bus 44. An input/output (I/O) interface 45 is also connected to bus 44.
Various components in electronic device 40 are connected to I/O interface 45, including: an input unit 46 such as a keyboard, a mouse, etc.; an output unit 47 such as various types of displays, speakers, and the like; a storage unit 48 such as a magnetic disk, an optical disk, or the like; and a communication unit 49 such as a network card, modem, wireless communication transceiver, etc. The communication unit 49 allows the electronic device 40 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.
The processor 41 may be various general and/or special purpose processing components with processing and computing capabilities. Some examples of processor 41 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, digital Signal Processors (DSPs), and any suitable processor, controller, microcontroller, etc. The processor 41 performs the respective methods and processes described above, for example, a distribution conversion method of data.
In some embodiments, the distribution conversion method of data may be implemented as a computer program tangibly embodied on a computer-readable storage medium, such as the storage unit 48. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 40 via the ROM 42 and/or the communication unit 49. When the computer program is loaded into the RAM 43 and executed by the processor 41, one or more steps of the distribution conversion method of data described above may be performed. Alternatively, in other embodiments, the processor 41 may be configured to perform the distribution conversion method of the data in any other suitable manner (e.g., by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.
A computer program for carrying out methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be implemented. The computer program may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which a user can provide input to the electronic device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), blockchain networks, and the internet.
The computing system may include clients and servers. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical hosts and VPS service are overcome.
It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps described in the present invention may be performed in parallel, sequentially, or in a different order, so long as the desired results of the technical solution of the present invention are achieved, and the present invention is not limited herein.
The above embodiments do not limit the scope of the present invention. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of the present invention.
Claims (10)
1. A distribution conversion method of data, comprising:
acquiring configuration task parameters according to configuration information input operation aiming at a preset configuration page;
acquiring data source information according to the configuration task parameters, and displaying a data table corresponding to the data source information in the preset configuration page;
acquiring data conversion configuration information according to the modification operation on the data table, and acquiring a configuration file in a preset data format according to the configuration task parameters and the data conversion configuration information;
And generating a distribution conversion task according to the configuration file, and executing the distribution conversion task to realize distribution conversion of data.
2. The method of claim 1, wherein the configuration task parameters include operation parameter information and target database information, and obtaining a configuration file in a preset data format according to the configuration task parameters and the data conversion configuration information includes:
and splicing the operation parameter information, the data source information, the data conversion configuration information and the target database information to generate a configuration file with a preset data format.
3. The method of claim 2, wherein generating a distribution transformation task from the configuration file comprises:
according to the operation parameter information, carrying out operation parameter configuration on a preset calculation engine to obtain a target calculation engine;
and generating a distribution conversion task according to the data source information, the data conversion configuration information and the target database information through the target calculation engine.
4. The method of claim 2, wherein performing the distribution transformation task to effect distribution transformation of data comprises:
And applying for obtaining a plurality of task execution nodes through a preset task scheduling frame, and executing the distribution conversion task through each task execution node so as to realize the distribution conversion of data.
5. The method of claim 4, wherein executing the distribution transformation task by each of the task execution nodes to effect distribution transformation of data comprises:
acquiring a data table corresponding to the data source information according to the distribution conversion task through each task execution node;
converting the data table according to the data conversion configuration information to obtain a conversion data table;
and storing the conversion data table into a target database corresponding to the target database information.
6. The method according to any of claims 1-5, wherein the data conversion configuration information comprises read data information comprising an original column name, an original column type, an original column length and/or a default value and write data information comprising a modified column name, a modified column type, a modified column length and/or a data conversion function;
wherein the data transfer function includes at least one of a field cut function, a field aggregate function, a field statistics function, and a date transfer function.
7. The method of any one of claims 1-5, wherein the operating parameter information includes at least one of an operating mode, an application name, an actuator core number, and a memory; the data source information comprises a data source type and data source connection information, and the data source connection information comprises at least one of a user name, a password, an access path, a database identifier and a data table identifier; the target database information includes at least one of a target library type, target library connection information, and a write mode.
8. A data distribution conversion apparatus, comprising:
the configuration task parameter acquisition module is used for acquiring configuration task parameters according to configuration information input operation aiming at a preset configuration page;
the data table display module is used for acquiring data source information according to the configuration task parameters and displaying a data table corresponding to the data source information in the preset configuration page;
the configuration file acquisition module is used for acquiring data conversion configuration information according to the modification operation on the data table, and acquiring a configuration file in a preset data format according to the configuration task parameters and the data conversion configuration information;
And the distribution conversion task generation module is used for generating a distribution conversion task according to the configuration file and realizing distribution conversion of data by executing the distribution conversion task.
9. An electronic device, the electronic device comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,,
the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the distribution conversion method of data of any one of claims 1-7.
10. A computer readable storage medium storing computer instructions for causing a processor to implement the method of distribution conversion of data according to any one of claims 1 to 7 when executed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310629208.9A CN116821217A (en) | 2023-05-30 | 2023-05-30 | Data distribution conversion method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310629208.9A CN116821217A (en) | 2023-05-30 | 2023-05-30 | Data distribution conversion method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116821217A true CN116821217A (en) | 2023-09-29 |
Family
ID=88111875
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310629208.9A Pending CN116821217A (en) | 2023-05-30 | 2023-05-30 | Data distribution conversion method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116821217A (en) |
-
2023
- 2023-05-30 CN CN202310629208.9A patent/CN116821217A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN115146000A (en) | Database data synchronization method and device, electronic equipment and storage medium | |
| CN111352951A (en) | Data export method, device and system | |
| CN114428674A (en) | Task scheduling method, device, equipment and storage medium | |
| CN112925811B (en) | Method, apparatus, device, storage medium and program product for data processing | |
| CN116009847A (en) | Code generation method, device, electronic equipment and storage medium | |
| CN116126719A (en) | Interface testing method and device, electronic equipment and storage medium | |
| CN115905322A (en) | Service processing method and device, electronic equipment and storage medium | |
| CN115454971A (en) | Data migration method and device, electronic equipment and storage medium | |
| CN115438056A (en) | Data acquisition method, device, equipment and storage medium | |
| CN116821217A (en) | Data distribution conversion method, device, equipment and storage medium | |
| CN113779117A (en) | Data monitoring method and device, storage medium and electronic equipment | |
| CN116431698B (en) | Data extraction method, device, equipment and storage medium | |
| CN116579914B (en) | Execution method and device of graphic processor engine, electronic equipment and storage medium | |
| CN115967638A (en) | Equipment simulation system, method, equipment and storage medium | |
| CN115794131A (en) | Method and device for automatically generating thermal deployment program and electronic equipment | |
| CN114706578A (en) | Data processing method, device, equipment and medium | |
| CN117331475A (en) | Task creation method, device, equipment and storage medium | |
| CN115168760A (en) | Data query method, device and storage medium | |
| CN117271104A (en) | Resource arrangement method, device, electronic equipment and storage medium | |
| CN117193726A (en) | Parallel design method and device of software, electronic equipment and medium | |
| CN115544418A (en) | Webpage data synchronization method and device, electronic equipment and storage medium | |
| CN115686640A (en) | Pipeline information processing method, device, equipment and storage medium | |
| CN117709902A (en) | Material input method, device, equipment and medium based on BOM file | |
| CN115665256A (en) | Task processing method and device, electronic equipment and storage medium | |
| CN116755744A (en) | Patch package generation method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Country or region after: China Address after: 100193 5 floor, 36 building, No. 8 Northeast Road, Haidian District, Beijing. Applicant after: Shuguang Cloud Computing Group Co.,Ltd. Address before: 100193 5 floor, 36 building, No. 8 Northeast Road, Haidian District, Beijing. Applicant before: Shuguang Cloud Computing Group Co.,Ltd. Country or region before: China |
|
| CB02 | Change of applicant information |