CN117435589A - Data transfer method, device, computer equipment and storage medium - Google Patents
Data transfer method, device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN117435589A CN117435589A CN202311278125.6A CN202311278125A CN117435589A CN 117435589 A CN117435589 A CN 117435589A CN 202311278125 A CN202311278125 A CN 202311278125A CN 117435589 A CN117435589 A CN 117435589A
- Authority
- CN
- China
- Prior art keywords
- data
- partition
- target
- field
- determining
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012546 transfer Methods 0.000 title claims abstract description 101
- 238000000034 method Methods 0.000 title claims abstract description 41
- 238000005192 partition Methods 0.000 claims abstract description 347
- 238000004590 computer program Methods 0.000 claims abstract description 29
- 238000013500 data storage Methods 0.000 claims description 35
- 238000012545 processing Methods 0.000 claims description 7
- 238000013075 data extraction Methods 0.000 claims description 4
- 238000004891 communication Methods 0.000 description 8
- 230000004044 response Effects 0.000 description 5
- 238000011161 development Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000012216 screening Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- OKTJSMMVPCPJKN-UHFFFAOYSA-N Carbon Chemical compound [C] OKTJSMMVPCPJKN-UHFFFAOYSA-N 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 229910021389 graphene Inorganic materials 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application relates to a data transfer method, a data transfer device, computer equipment, a storage medium and a computer program product, relates to the technical field of big data, and can improve data transfer efficiency. The method comprises the following steps: responsive to a data-dumping request to dump data in a data warehouse to an unstructured database, exposing candidate partition fields for characterizing the data warehouse partition attribute information; determining a target partition field based on a selected operation for the candidate partition field; determining a target data partition associated with the target partition field in a plurality of data partitions of the data warehouse, and reading data under the target data partition; and transferring the data under the target data partition to the unstructured database.
Description
Technical Field
The present invention relates to the field of big data technology, and in particular, to a data transfer method, an apparatus, a computer device, a storage medium, and a computer program product.
Background
With the development of financial technology, data processed by financial service applications grows exponentially, and how to effectively store and calculate massive data every day is a urgent problem to be solved by the financial service applications.
In the related art, when data in a data warehouse is transferred to an unstructured database, after determining that a field corresponding to the transferred data is required, the data warehouse is traversed, and the data corresponding to the field is queried and transferred to the unstructured database.
However, the foregoing manner often needs to wait for a period of time before the transfer can be completed, which has a problem of low efficiency of the data transfer manner.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a data transfer method, apparatus, computer device, computer readable storage medium, and computer program product that can improve the data transfer efficiency.
In a first aspect, the present application provides a data transfer method. The method comprises the following steps:
responsive to a data-dumping request to dump data in a data warehouse to an unstructured database, exposing candidate partition fields for characterizing the data warehouse partition attribute information;
determining a target partition field based on a selected operation for the candidate partition field;
determining a target data partition associated with the target partition field in a plurality of data partitions of the data warehouse, and reading data under the target data partition;
and transferring the data under the target data partition to the unstructured database.
In one embodiment, the transferring the data under the target data partition to the unstructured database includes:
determining a data storage structure of a database table in the unstructured database; the database table is used for storing data under the target data partition;
and storing the data under the target data partition into the database table according to the table structure.
In one embodiment, the determining the data storage structure of the database table in the unstructured database includes:
determining a data storage structure preconfigured for each data partition;
and acquiring a data storage structure corresponding to the target data partition from the pre-configured data storage structure, and taking the data storage structure as a data storage structure of a database table in the unstructured database.
In one embodiment, the determining, among the plurality of data partitions of the data warehouse, a target data partition associated with the target partition field includes:
acquiring partition field indexes corresponding to a plurality of data partitions of the data warehouse;
and determining a partition field index containing the target partition field from the partition field indexes, and taking the data partition corresponding to the partition field index containing the target partition field as a target data partition.
In one embodiment, before the presenting of the candidate partition fields for characterizing the data warehouse partition attribute information, further comprises:
acquiring data transfer frequencies corresponding to a plurality of data partitions in the data warehouse; the data transfer frequency represents the frequency of data transfer processing of the data in the corresponding data partition;
and determining at least one high-frequency transfer partition with highest data transfer frequency from the plurality of data partitions, and obtaining the candidate partition field to be displayed based on the partition field of the at least one high-frequency transfer partition.
In one embodiment, before the presenting of the candidate partition fields for characterizing the data warehouse partition attribute information, further comprises:
acquiring a task type of a data transfer task aimed at by the data transfer request;
and determining a necessary data partition corresponding to the task type, and obtaining candidate partition fields to be displayed based on the partition fields of the necessary data partition.
In a second aspect, the present application further provides a data transfer device. The device comprises:
a candidate partition field display module, configured to respond to a data transfer request for transferring data in a data warehouse to an unstructured database, and display a candidate partition field for characterizing partition attribute information of the data warehouse;
a target partition field determining module, configured to determine a target partition field based on a selected operation for the candidate partition field;
the data extraction module is used for determining a target data partition associated with the target partition field in a plurality of data partitions of the data warehouse and reading data under the target data partition;
and the data transfer module is used for transferring the data under the target data partition to the unstructured database.
In a third aspect, the present application also provides a computer device. The computer device comprises a memory storing a computer program and a processor which when executing the computer program performs the steps of:
responsive to a data-dumping request to dump data in a data warehouse to an unstructured database, exposing candidate partition fields for characterizing the data warehouse partition attribute information;
determining a target partition field based on a selected operation for the candidate partition field;
determining a target data partition associated with the target partition field in a plurality of data partitions of the data warehouse, and reading data under the target data partition;
and transferring the data under the target data partition to the unstructured database.
In a fourth aspect, the present application also provides a computer-readable storage medium. The computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of:
responsive to a data-dumping request to dump data in a data warehouse to an unstructured database, exposing candidate partition fields for characterizing the data warehouse partition attribute information;
determining a target partition field based on a selected operation for the candidate partition field;
determining a target data partition associated with the target partition field in a plurality of data partitions of the data warehouse, and reading data under the target data partition;
and transferring the data under the target data partition to the unstructured database.
In a fifth aspect, the present application also provides a computer program product. The computer program product comprises a computer program which, when executed by a processor, implements the steps of:
responsive to a data-dumping request to dump data in a data warehouse to an unstructured database, exposing candidate partition fields for characterizing the data warehouse partition attribute information;
determining a target partition field based on a selected operation for the candidate partition field;
determining a target data partition associated with the target partition field in a plurality of data partitions of the data warehouse, and reading data under the target data partition;
and transferring the data under the target data partition to the unstructured database.
In the above data transfer method, apparatus, computer device, storage medium and computer program product, in response to a data transfer request for transferring data in a data warehouse to an unstructured database, a candidate partition field for characterizing attribute information of the data warehouse partition may be displayed, and a target partition field is determined based on a selection operation for the candidate partition field, and then, among a plurality of data partitions in the structured database, a target data partition associated with the target partition field may be determined, and data under the target data partition may be read, and then transferred to the unstructured database. In this embodiment, on one hand, by displaying the candidate partition fields, a plurality of data partitions capable of performing data transfer can be provided to a user, so that the user is prevented from searching one by one, and on the other hand, by determining a corresponding target data partition in the data warehouse and transferring data under the target data partition to the unstructured database according to the target partition field determined by the selected operation, the whole data warehouse is prevented from being traversed when the data is transferred to the unstructured database, and the query speed of the data to be transferred is improved, so that the efficiency of transferring the data to the unstructured database is effectively improved.
Drawings
FIG. 1 is a flow chart illustrating a method of data transfer in one embodiment;
FIG. 2 is a flowchart illustrating steps for transferring data according to a table structure in one embodiment;
FIG. 3 is a flowchart of a data transfer method according to another embodiment;
FIG. 4 is a block diagram illustrating a data transfer device according to an embodiment;
FIG. 5 is an internal block diagram of a computer device in one embodiment;
FIG. 6 is an internal block diagram of another computer device in one embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present application.
In one embodiment, as shown in fig. 1, a data transfer method is provided, where the method is applied to a terminal to illustrate the method, it is understood that the method may also be applied to a server, and may also be applied to a system including the terminal and the server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the steps of:
s101, in response to a data transfer request for transferring data in a data warehouse to an unstructured database, a candidate partition field for representing the partition attribute information of the data warehouse is displayed.
The partition field may characterize partition attribute information of a database table in the data warehouse, which may include, for example, at least one of: the method comprises the steps of partition names corresponding to data partitions, creation time of the data partitions, storage time of data in the data partitions and storage addresses of the data partitions.
In some embodiments, the data warehouse may include a Hive data warehouse for financial business data. Hive is a data warehouse tool based on Hadoop, and can be used for extracting, converting and loading data, and is a mechanism capable of storing, querying and analyzing large-scale data stored in Hadoop. The Hive data warehouse tool can map data files in a structured database into a database table, such as a Hive database table, and provide SQL query functions, and can convert SQL sentences into MapReduce tasks to be executed. Hive can realize rapid MapReduce statistics through SQL-like sentences, so that MapReduce is simpler without developing a special MapReduce application program, and the method is suitable for carrying out statistical analysis on a data warehouse.
The unstructured database comprises an HBase database and the data in the unstructured database may comprise HBase tables. Wherein, the HBase is a distributed, column-oriented open source database, which can store data based on a column pattern.
In some embodiments, the user may trigger a data transfer request for transferring data in the financial service data warehouse to the unstructured database, for example, may click a preset transfer button in an application interface of the financial service data warehouse management application, and after detecting that the user performs the triggering operation on the button, the terminal may determine that the data transfer request for transferring data in the data warehouse to the unstructured database is received.
In response to the data transfer request, the candidate partition field may be displayed in the interface, so that the user may select a data partition for performing data transfer subsequently in a configuration manner, in other words, may perform configuration development on the candidate partition field in advance, for example, may determine a relevant partition field according to a historical data transfer task, and perform configuration development on the relevant partition field, where the configuration development may refer to setting a configuration item corresponding to the relevant partition field, and further may obtain corresponding configuration information and perform data transfer when the relevant partition field is selected.
In some alternative embodiments, hive's metadata may be obtained from MySQL, from which candidate partition fields are derived.
S102, determining a target partition field based on the selected operation for the candidate partition field.
After presenting one or more candidate partition fields, the user may perform a selection operation on at least one currently presented candidate partition field, and then the terminal may determine a target partition field according to the detected selection operation.
S103, determining a target data partition associated with the target partition field in a plurality of data partitions of the data warehouse, and reading data under the target data partition.
In a specific implementation, the database table in the data warehouse may be divided into a plurality of data partitions according to a preset partition rule, for example, the data partitions are performed according to a data type or a data storage time, and in a specific implementation, a worker may partition the database table according to one or more data features.
In some embodiments, the partition field may include a data attribute corresponding to the data stored in the data partition, which may be determined based on, for example, metadata stored in MySQL, such as a generation time, a storage address, a data type, and the like of the data. In other embodiments, the partition field may also be the name of the partition table.
In some alternative embodiments, where the data warehouse is a Hive data warehouse, the plurality of data partitions may be partition tables in a Hive database table.
In this step, after determining the target partition field, a target data partition associated with the target partition field may be determined among the plurality of database partitions, and data under the target data partition may be read.
In some possible embodiments, only data description information may be stored under the target data partition, for example, for the target data partition a, the description information of the data a (such as a file or a table) is stored, while specific data content of the data a may be stored in other databases, after the target data partition is determined, corresponding data may be queried according to the data description information stored in the target data partition, and the corresponding data may be used as the data under the target data partition.
S104, the data under the target data partition is transferred to an unstructured database.
After the data under the target data partition is obtained, the data may be transferred to an unstructured database.
In this embodiment, in response to a data transfer request for transferring data in a data warehouse to an unstructured database, a partition field that is a candidate for characterizing attribute information of the data warehouse partition may be displayed, and a target partition field is determined based on a selection operation for the candidate partition field, and then, among a plurality of data partitions in the structured database, a target data partition associated with the target partition field may be determined, and data under the target data partition may be read, and then, data under the target data partition may be transferred to the unstructured database. In this embodiment, on one hand, by displaying the candidate partition fields, a plurality of data partitions capable of performing data transfer can be provided to a user, so that the user is prevented from searching one by one, and on the other hand, by determining a corresponding target data partition in the data warehouse and transferring data under the target data partition to the unstructured database according to the target partition field determined by the selected operation, the whole data warehouse is prevented from being traversed when the data is transferred to the unstructured database, and the query speed of the data to be transferred is improved, so that the efficiency of transferring the data to the unstructured database is effectively improved.
In one embodiment, as shown in fig. 2, S104 may include the following steps of:
s201, determining a data storage structure of a database table in the unstructured database.
Wherein the database tables include database tables for storing data under the target data partition. In an embodiment, if the unstructured database is an HBase database, the database tables in the unstructured database may include HBase tables.
S202, storing the data under the target data partition into a database table according to the table structure.
For example, after the data under the target data partition is acquired, a data field corresponding to the data under the target data partition may be determined, where the data field may be understood as a feature used to compose a data record, and may also be referred to as a data item. After determining the data field corresponding to the data, the data field may be mapped to the corresponding field in the data storage structure one by one, and then the data under the target data partition may be stored in the database table.
In this embodiment, the data under the target data partition is orderly stored in the database table according to the data storage structure of the database table, so that the efficiency of querying the data in the unstructured database later can be improved, and the data utilization rate can be increased.
In one embodiment, step S201 determines the data storage structure of the database table in the unstructured database, and may include the steps of:
determining a data storage structure preconfigured for each data partition; and acquiring a data storage structure corresponding to the target data partition from the pre-configured data storage structure, and taking the data storage structure as a data storage structure of a database table in the unstructured database.
In practical application, the data in each data partition has relatively stable corresponding data fields, and the fields in the data partition and the fields in the database table in the unstructured database have association relation, so that for each data partition, the fields corresponding to the data in the data partition and the fields in the database table in the unstructured database can be mapped one by one to determine the fields in the data partition, and the fields corresponding to the unstructured database, for example, the fields in the Hive table and the fields in the Hbase table are mapped one by one. The corresponding data storage structure, which may also be referred to as a table structure, may then be configured according to the corresponding fields in the determined out-unstructured database.
And after determining the target data partition, the data storage structure corresponding to the target data partition can be obtained from a plurality of pre-configured data storage structures and used as the data storage structure of the database table in the unstructured database.
In this embodiment, by configuring the matched data storage structures for different data partitions, it is possible to ensure that data in subsequent different data partitions is transferred to the database table in the unstructured database in order.
In one embodiment, step S103, among the plurality of data partitions in the data warehouse, determining a target data partition associated with the target partition field may include the steps of:
acquiring partition field indexes corresponding to a plurality of data partitions of a data warehouse; and determining a partition field index containing the target partition field from the partition field indexes, and taking the data partition corresponding to the partition field index containing the target partition field as a target data partition.
In practical application, after the data partition is created in the data warehouse, a corresponding partition field index can be generated according to the partition field in the currently created data partition, so that the partition field indexes corresponding to the data partitions can be obtained, wherein the partition field indexes comprise partition fields corresponding to the data partition.
Then, after the target partition field is obtained, the partition field index including the target partition field may be queried in the multiple partition field indexes, and if the target partition field is multiple partition fields for the same data partition, the partition field indexes including the multiple partition fields may be queried in the multiple partition field indexes. Therefore, when the data to be transferred in the data warehouse is determined, the whole data warehouse or each data partition is prevented from being traversed, the target data partition is rapidly positioned, and the data transfer efficiency is effectively improved.
In one embodiment, prior to exposing the candidate partition fields for characterizing the data warehouse partition attribute information, the method may further comprise the steps of:
acquiring data transfer frequencies corresponding to a plurality of data partitions in a data warehouse; and determining at least one high-frequency transfer partition with highest data transfer frequency from the plurality of data partitions, and obtaining the candidate partition fields to be displayed based on the partition fields of the at least one high-frequency transfer partition.
The data transfer frequency represents the frequency of data transfer processing of the data in the corresponding data partition.
Specifically, for example, the data transfer times of the data partitions may be counted to obtain the data transfer frequency of each data partition, and in an alternative embodiment, the statistics may be performed in a dimension of each user, that is, when the user triggers the data transfer request, the data transfer frequency of the user for each data partition is obtained. At least one data partition with the highest data transfer frequency can then be used as the high-frequency transfer partition.
In this embodiment, the partition field of the data partition which may be subjected to data transfer can be provided to the user by using the partition field of at least one high-frequency transfer partition as the partition field of the candidate to be displayed, so that the time for the user to find the target partition field is reduced, and the efficiency of data transfer is improved.
In one embodiment, prior to exposing the candidate partition fields for characterizing the data warehouse partition attribute information, the method may further comprise the steps of:
acquiring a task type of a data transfer task aimed at by a data transfer request; and determining a necessary data partition corresponding to the task type, and obtaining a candidate partition field to be displayed based on the partition field of the necessary data partition.
In one embodiment, a user may create a data transfer task and set a task type of the data transfer task, where data to be transferred corresponding to different task types may be preset. For example, for a data transfer task of bill information, the data to be transferred may include historical bill information of the user within a preset time, and for a business record transfer task of the target financial business, the data to be transferred includes a business record of the target financial business.
Then, the task type of the data transfer task can be determined according to the data transfer request. For example, a task type identifier of the data transfer task may be read from the data transfer request, and a task type corresponding to the data transfer task to be executed is determined according to the task type identifier. Further, a necessary data partition corresponding to the task type may be determined, where the necessary data partition may be understood as a data partition that needs to be accessed and data is restored when the current data restoring task is performed, for example, for a data restoring task for billing information, the necessary data partition may be a data partition corresponding to a specified history time.
In this embodiment, by determining the partition field of the necessary data partition as the candidate partition field, the time for the user to find the target partition field can be reduced, the correct partition field can be provided for the user to confirm, and the data transfer efficiency is improved.
In order to enable those skilled in the art to better understand the above steps, the embodiments of the present application will be exemplified below by way of an example, but it should be understood that the embodiments of the present application are not limited thereto.
As shown in fig. 3, in practical application, metadata configured for the Hive table may be obtained from MySQL, and according to the metadata, a plurality of configurable candidate partition fields are presented to the user. Then, the user's selection operation for the candidate partition field may be followed, the selected partition field is determined as the target partition field, and the target data partition corresponding to the target partition field is determined in Hive.
The corresponding data may then be extracted from the target data partition, which in some alternative embodiments may be extracted in conjunction with the configured screening conditions, e.g., data matching the screening conditions may be extracted from the target data partition.
Then, the fields in the Hive partition table and the fields in the Hbase table may be mapped one by one, for example, the columns in the Hbase table and the columns in the Hive table are mapped one by one in sequence, the fields in the unstructured database associated with the extracted data are determined, and the table structure is determined according to the associated fields, so that the extracted data may be stored in the Hbase table according to the table structure.
It should be understood that, although the steps in the flowcharts related to the embodiments described above are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.
Based on the same inventive concept, the embodiment of the application also provides a data transfer device for realizing the above related data transfer method. The implementation of the solution provided by the device is similar to the implementation described in the above method, so the specific limitation in one or more embodiments of the data transfer device provided below may refer to the limitation of the data transfer method hereinabove, and will not be described herein.
In one embodiment, as shown in fig. 4, there is provided a data transfer device, including:
a candidate partition field presentation module 401, configured to, in response to a data transfer request for transferring data in a data warehouse to an unstructured database, present a candidate partition field for characterizing partition attribute information of the data warehouse;
a target partition field determination module 402, configured to determine a target partition field based on a selected operation for the candidate partition field;
a data extraction module 403, configured to determine, among a plurality of data partitions in the data warehouse, a target data partition associated with the target partition field, and read data under the target data partition;
and the data transfer module 404 is configured to transfer the data in the target data partition to the unstructured database.
In one embodiment, the data transfer module 404 is configured to:
determining a data storage structure of a database table in the unstructured database; the database table is used for storing data under the target data partition;
and storing the data under the target data partition into the database table according to the table structure.
In one embodiment, the data transfer module 404 is configured to:
determining a data storage structure preconfigured for each data partition;
and acquiring a data storage structure corresponding to the target data partition from the pre-configured data storage structure, and taking the data storage structure as a data storage structure of a database table in the unstructured database.
In one embodiment, the data extraction module 403 is configured to:
acquiring partition field indexes corresponding to a plurality of data partitions of the data warehouse;
and determining a partition field index containing the target partition field from the partition field indexes, and taking the data partition corresponding to the partition field index containing the target partition field as a target data partition.
In one embodiment, the candidate partition field exhibition module 401 is configured to:
acquiring data transfer frequencies corresponding to a plurality of data partitions in the data warehouse; the data transfer frequency represents the frequency of data transfer processing of the data in the corresponding data partition;
and determining at least one high-frequency transfer partition with highest data transfer frequency from the plurality of data partitions, and obtaining the candidate partition field to be displayed based on the partition field of the at least one high-frequency transfer partition.
In one embodiment, the candidate partition field exhibition module 401 is configured to:
acquiring a task type of a data transfer task aimed at by the data transfer request;
and determining a necessary data partition corresponding to the task type, and obtaining candidate partition fields to be displayed based on the partition fields of the necessary data partition.
The modules in the data transfer device may be implemented in whole or in part by software, hardware, or a combination thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
In one embodiment, a computer device is provided, which may be a server, the internal structure of which may be as shown in fig. 5. The computer device includes a processor, a memory, an Input/Output interface (I/O) and a communication interface. The processor, the memory and the input/output interface are connected through a system bus, and the communication interface is connected to the system bus through the input/output interface. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device is used for storing data for realizing the data transfer method. The input/output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a data transfer method.
In one embodiment, a computer device is provided, which may be a terminal, and the internal structure of which may be as shown in fig. 6. The computer device includes a processor, a memory, an input/output interface, a communication interface, a display unit, and an input means. The processor, the memory and the input/output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input/output interface. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The input/output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used for carrying out wired or wireless communication with an external terminal, and the wireless mode can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. The computer program is executed by a processor to implement a data transfer method. The display unit of the computer device is used for forming a visual picture, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, can also be a key, a track ball or a touch pad arranged on the shell of the computer equipment, and can also be an external keyboard, a touch pad or a mouse and the like.
It will be appreciated by those skilled in the art that the structures shown in fig. 5 and 6 are block diagrams of only portions of structures that are relevant to the present application and are not intended to limit the computer device on which the present application may be implemented, and that a particular computer device may include more or fewer components than shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a computer device is provided, comprising a memory and a processor, the memory having stored therein a computer program, the processor implementing the steps of the method embodiments described above when the computer program is executed.
In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored which, when executed by a processor, implements the steps of the method embodiments described above.
In an embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of the method embodiments described above.
It should be noted that, the user information (including, but not limited to, user equipment information, user personal information, etc.) and the data (including, but not limited to, data for analysis, stored data, presented data, etc.) referred to in the present application are information and data authorized by the user or sufficiently authorized by each party, and the collection, use and processing of the related data are required to comply with the related laws and regulations and standards of the related countries and regions.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in the various embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (ReRAM), magnetic random access Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (Phase Change Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in the form of a variety of forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), and the like. The databases referred to in the various embodiments provided herein may include at least one of relational databases and non-relational databases. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processors referred to in the embodiments provided herein may be general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic units, quantum computing-based data processing logic units, etc., without being limited thereto.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The above examples only represent a few embodiments of the present application, which are described in more detail and are not to be construed as limiting the scope of the present application. It should be noted that it would be apparent to those skilled in the art that various modifications and improvements could be made without departing from the spirit of the present application, which would be within the scope of the present application. Accordingly, the scope of protection of the present application shall be subject to the appended claims.
Claims (10)
1. A method of data transfer, the method comprising:
responsive to a data-dumping request to dump data in a data warehouse to an unstructured database, exposing candidate partition fields for characterizing the data warehouse partition attribute information;
determining a target partition field based on a selected operation for the candidate partition field;
determining a target data partition associated with the target partition field in a plurality of data partitions of the data warehouse, and reading data under the target data partition;
and transferring the data under the target data partition to the unstructured database.
2. The method of claim 1, wherein the transferring data under the target data partition to the unstructured database comprises:
determining a data storage structure of a database table in the unstructured database; the database table is used for storing data under the target data partition;
and storing the data under the target data partition into the database table according to the table structure.
3. The method of claim 2, wherein said determining a data storage structure of a database table in said unstructured database comprises:
determining a data storage structure preconfigured for each data partition;
and acquiring a data storage structure corresponding to the target data partition from the pre-configured data storage structure, and taking the data storage structure as a data storage structure of a database table in the unstructured database.
4. The method of claim 1, wherein the determining, among the plurality of data partitions of the data warehouse, a target data partition associated with the target partition field comprises:
acquiring partition field indexes corresponding to a plurality of data partitions of the data warehouse;
and determining a partition field index containing the target partition field from the partition field indexes, and taking the data partition corresponding to the partition field index containing the target partition field as a target data partition.
5. The method of claim 1, further comprising, prior to said exposing candidate partition fields for characterizing said data warehouse partition attribute information:
acquiring data transfer frequencies corresponding to a plurality of data partitions in the data warehouse; the data transfer frequency represents the frequency of data transfer processing of the data in the corresponding data partition;
and determining at least one high-frequency transfer partition with highest data transfer frequency from the plurality of data partitions, and obtaining the candidate partition field to be displayed based on the partition field of the at least one high-frequency transfer partition.
6. The method of any of claims 1-5, further comprising, prior to said exposing a partition field for characterizing candidates of the data warehouse partition attribute information:
acquiring a task type of a data transfer task aimed at by the data transfer request;
and determining a necessary data partition corresponding to the task type, and obtaining candidate partition fields to be displayed based on the partition fields of the necessary data partition.
7. A data transfer device, the device comprising:
a candidate partition field display module, configured to respond to a data transfer request for transferring data in a data warehouse to an unstructured database, and display a candidate partition field for characterizing partition attribute information of the data warehouse;
a target partition field determining module, configured to determine a target partition field based on a selected operation for the candidate partition field;
the data extraction module is used for determining a target data partition associated with the target partition field in a plurality of data partitions of the data warehouse and reading data under the target data partition;
and the data transfer module is used for transferring the data under the target data partition to the unstructured database.
8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any of claims 1 to 6 when the computer program is executed.
9. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 6.
10. A computer program product comprising a computer program, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311278125.6A CN117435589A (en) | 2023-09-28 | 2023-09-28 | Data transfer method, device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311278125.6A CN117435589A (en) | 2023-09-28 | 2023-09-28 | Data transfer method, device, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117435589A true CN117435589A (en) | 2024-01-23 |
Family
ID=89556084
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311278125.6A Pending CN117435589A (en) | 2023-09-28 | 2023-09-28 | Data transfer method, device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117435589A (en) |
-
2023
- 2023-09-28 CN CN202311278125.6A patent/CN117435589A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9875272B1 (en) | Method and system for designing a database system for high event rate, while maintaining predictable query performance | |
| US9773010B1 (en) | Information-driven file system navigation | |
| CN115544183B (en) | Data visualization method, device, computer equipment and storage medium | |
| WO2024187779A1 (en) | Service data storage method and apparatus, computer device, and storage medium | |
| WO2024016789A1 (en) | Log data query method and apparatus, and device and medium | |
| CN117390011A (en) | Report data processing method, device, computer equipment and storage medium | |
| CN111008198A (en) | Service data acquisition method and device, storage medium and electronic equipment | |
| CN114547066A (en) | Nuclear power business data standardization method and device and computer equipment | |
| CN117435589A (en) | Data transfer method, device, computer equipment and storage medium | |
| CN117290302B (en) | Directory separation method, apparatus, computer device and storage medium | |
| CN114781346B (en) | Audit model sharing method, audit model sharing device, audit model sharing computer equipment and audit model sharing computer program product | |
| CN117909550A (en) | Query method, query device, computer equipment and storage medium | |
| CN112579605B (en) | Data storage method, device, storage medium and server | |
| CN118132656A (en) | Knowledge graph construction method, knowledge graph construction device, computer equipment and storage medium | |
| CN118331961A (en) | Method and device for storing target account in separate storage, computer equipment and readable storage medium | |
| CN117648336A (en) | Data query method, device, computer equipment and storage medium | |
| CN117725266A (en) | Load curve data processing method and device and intelligent ammeter | |
| CN118245699A (en) | Page acquisition method, device, equipment, storage medium and computer program product | |
| CN118332002A (en) | Data query method, device, computer equipment and storage medium | |
| CN116775955A (en) | Data tag solidifying method, device, computer equipment and storage medium | |
| CN118152622A (en) | Processing method, device, equipment, medium and product of data asset directory tree | |
| CN114610688A (en) | Log aggregation method and device, computer equipment and storage medium | |
| CN118152504A (en) | Unstructured data indexing method, device, apparatus, medium and program product | |
| CN118503269A (en) | Data searching method, device, computer equipment and readable storage medium | |
| CN118296059A (en) | Resource processing method, apparatus, computer device, storage medium, and program product |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |