CN114925074A - Data unloading method, device, equipment and medium - Google Patents
Data unloading method, device, equipment and medium Download PDFInfo
- Publication number
- CN114925074A CN114925074A CN202210706934.1A CN202210706934A CN114925074A CN 114925074 A CN114925074 A CN 114925074A CN 202210706934 A CN202210706934 A CN 202210706934A CN 114925074 A CN114925074 A CN 114925074A
- Authority
- CN
- China
- Prior art keywords
- unloading
- database table
- time
- mode
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 43
- 238000004590 computer program Methods 0.000 claims description 14
- 238000012545 processing Methods 0.000 abstract description 8
- 238000007405 data analysis Methods 0.000 description 18
- 230000000694 effects Effects 0.000 description 10
- 230000008569 process Effects 0.000 description 5
- 230000008676 import Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000007619 statistical method Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000033228 biological regulation Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
Abstract
The invention relates to the technical field of data processing, in particular to a data unloading method, a data unloading device, data unloading equipment and a data unloading medium. The method specifically comprises the following steps: generating a database table list to be unloaded according to a plurality of database tables, wherein the database table list comprises respective indexes of the plurality of database tables and a plurality of field information, and the plurality of field information comprises field information corresponding to each database table in the plurality of database tables; generating unloading parameters according to field information of the database table list, wherein the unloading parameters are used for unloading a plurality of database tables; and importing the unloading parameters into an unloading script to generate an unloading file, wherein the unloading script is used for unloading the plurality of database tables. The method can dynamically configure the unloading time of each database table, thereby unloading the data in different data sources simultaneously and improving the unloading efficiency.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to a data unloading method, a data unloading device, data unloading equipment and a data unloading medium.
Background
In recent years, with the dramatic increase in information volume, a large amount of data is generated along with daily life. For example, when the host develops a marketing campaign, there are data indicating the participation of the campaign, data indicating the development of the campaign, data indicating the issuance of a prize, and the like. Statistical analysis of these data is often required. Because the data are from different sources, statistical analysis is performed on the data, and the data are generally stored in a database to generate a database table. And sending the database table to a data analysis system for analysis. The most common approach is to use Oracle to generate an offload file for offloading the database tables before sending it to the data analysis system. In this way, the data analysis system receives the unload file and reloads the unload file to obtain the data in the database table.
Currently, only one database table is unloaded (i.e. processing, such as extracting partial data) at a time, and in a scene of multiple data sources, the unloading efficiency is low. And since different data sources have different service execution dates, the different data sources are unloaded uniformly, and unnecessary data can be unloaded.
Disclosure of Invention
Embodiments of the present invention provide a data offloading method, apparatus, device, and medium, which can offload data in different data sources simultaneously when different data sources exist, so as to improve data offloading efficiency.
In a first aspect, an embodiment of the present invention provides a data offloading method, which may be performed by an electronic device with computing processing capability, for example, a personal computer, and includes:
the electronic equipment generates a database table list to be unloaded according to the plurality of database tables, wherein the database table list comprises respective indexes and a plurality of field information of the plurality of database tables, and the plurality of field information comprises the field information corresponding to each database table in the plurality of database tables. Then, the electronic equipment generates unloading parameters according to the field information of the database table list, and the unloading parameters are used for unloading a plurality of database tables. And then, the electronic equipment imports the unloading parameters into an unloading script to generate an unloading file, wherein the unloading script is used for unloading the plurality of database tables. In the scene of multiple data sources, when data in different data sources are unloaded, different data sources have different unloading dates, and unnecessary data may be unloaded by uniformly unloading different data sources. Therefore, in the embodiment of the present invention, the offloading time of the data source may be dynamically configured, for example, the offloading time may be the same for different data sources. And when unloading is required, generating a database table list according to the database tables of different data sources to be unloaded, and generating an unloading file according to unloading parameters generated by the database table list. Therefore, the unloading file is operated, and each database table in the database table list can be unloaded in sequence according to the unloading parameters, namely, the simultaneous unloading of multiple data sources is realized, and the unloading efficiency is improved.
Optionally, the multiple pieces of field information of the database table list include unloading mode information, the unloading mode information includes an unloading mode, the unloading mode includes a full unloading mode or an incremental unloading mode, the full unloading mode indicates to unload data in all the database tables included in the database table list, the incremental unloading mode indicates to unload data added in a preset time interval in the multiple database tables, and generating the unloading parameter according to the field information of the database table list includes: and the electronic equipment acquires the unloading mode information and generates the unloading parameters according to the unloading mode information. Therefore, when a plurality of database tables exist in the database table list, a user can configure different unloading modes for each database table according to actual requirements, and correspondingly, the electronic equipment generates unloading parameters according to the unloading modes configured by the user, so that flexible unloading of the database tables can be realized.
Optionally, the multiple pieces of field information of the database table list further include unloading time information, where the unloading time information indicates unloading time of the database table in an incremental unloading mode, and the generating of the unloading parameter according to the field information of the database table list includes: the electronic equipment acquires unloading time information and generates unloading parameters according to the unloading time information, wherein the unloading parameters comprise ith unloading time and a preset time interval, and the preset time interval and the ith unloading time are used for determining the (i + 1) th unloading time. It is understood that the incremental unloading mode unloads data within a predetermined time interval compared to the full unloading mode. Thus, the unload time for each database table may be dynamically configured. Different database tables may be set to the same unload time so that multiple database tables may be unloaded during the same period of time. In addition, the unloading time may include an ith unloading time and an i +1 th unloading time. The (i + 1) th unloading time can be calculated through the (i) th unloading time and a preset time interval, namely the (i + 1) th unloading time is indirectly configured through the preset time interval, the (i + 1) th unloading time does not need to be configured independently, the number of times of configuring the (i + 1) th unloading time can be reduced, and the method is simpler and more convenient.
Optionally, after the uninstalling parameters are imported into the uninstalling script to generate an uninstalling file, the method further includes: and the electronic equipment updates the (i + 1) th unloading time to the (i) th unloading time, and then acquires the (i) th unloading time and a preset time interval to generate new (i + 1) th unloading time. And the preset time intervals corresponding to different database tables are different. The scheme can update the (i + 1) th unloading time to the ith unloading time, so that the aim of circularly setting the unloading time of the database table can be fulfilled. Namely, the ith unloading time and the (i + 1) th unloading time can be circularly set by setting the ith unloading time and the preset time interval. At the same time, the unloading of each database table can be realized to be advanced along with time.
Optionally, the multiple pieces of field information of the database table list include an uninstallation identifier, where the uninstallation identifier is used to indicate whether the uninstallation of the database table is completed, and the method further includes: the electronic equipment acquires an unloading identifier corresponding to the first database table, and acquires push information if the unloading identifier indicates that the first database table is unloaded completely, wherein the push information comprises a storage path and a target file path of the first database table. Therefore, the electronic equipment acquires the first database table from the storage path and pushes the first database table to the target file path. In the embodiment of the invention, after the electronic device unloads the database table, the unloaded database table can be pushed. For example, the electronic device can determine which database tables are unloaded according to the unloading identifier, and then select the pushing time according to the requirement, and when the pushing time is reached, the database tables to be pushed can be obtained according to the pushing information, and the database tables are pushed to the target file path.
Optionally, the pushing information further includes a pushing mode identifier, where the pushing mode identifier indicates a pushing mode, and the electronic device pushes the first database to the target file path, including: the electronic equipment acquires the pushing mode identification and pushes the first database table to the target file path according to the pushing mode indicated by the pushing mode identification. In the embodiment of the present invention, there are multiple push modes, for example, a jfserver push mode, an ftp push mode, and an httpclient push mode. In different application scenes, a matched push mode can be selected, and the application range is wider.
Optionally, after the uninstalling parameters are imported into the uninstalling script to generate an uninstalling file, the method further includes: the electronic equipment receives the input unloading times and unloads the database table reaching the unloading time according to the unloading times. The user can set the unloading times according to actual requirements, and the electronic equipment can unload the data in a plurality of preset time intervals by setting the unloading times because the electronic equipment only unloads the data in the database table for one preset time interval at a time, so that the unloading efficiency of the electronic equipment is improved. Meanwhile, by setting the unloading times, when the unloading of the database table fails at a certain time, the unloading can be carried out next time, and data omission caused by the unloading failure is effectively avoided.
In a second aspect, an embodiment of the present invention provides a data offloading device, where the data modifying device may be configured to implement the data offloading method provided in the first aspect. The data unloading device comprises a list generation module, a parameter generation module and an unloading module. The list generation module is used for generating a database list to be unloaded according to a plurality of database tables, the database list comprises respective indexes of the plurality of database tables and a plurality of field information, and the plurality of field information comprises the field information corresponding to each database table in the plurality of database tables. And the parameter generation module is used for generating unloading parameters according to the field information of the database table list, and the unloading parameters are used for unloading the plurality of database tables. And the unloading module is used for importing the unloading parameters into an unloading script to generate an unloading file, and the unloading script is used for unloading the plurality of database tables.
Optionally, the parameter generating module is specifically configured to: and acquiring the unloading mode information, and generating the unloading parameters according to the unloading mode information.
Optionally, the parameter generating module is further configured to: acquiring unloading time information, and generating unloading parameters according to the unloading time information, wherein the unloading parameters comprise ith unloading time and a preset time interval, and the preset time interval and the ith unloading time are used for determining the (i + 1) th unloading time.
Optionally, the parameter generating module is further configured to: and updating the i +1 st unloading time to the i unloading time. And acquiring the ith unloading time and a preset time interval, and generating new (i + 1) th unloading time, wherein the preset time intervals corresponding to different database tables are different.
Optionally, the data offloading device is further configured to: and acquiring an unloading identifier corresponding to the first database table, and acquiring push information if the unloading identifier indicates that the first database table is unloaded, wherein the push information comprises a storage path and a target file path of the first database table. And acquiring the first database table from the storage path, and pushing the first database table to the target file path.
Optionally, the data offloading device is further configured to: and acquiring a push mode identifier, and pushing the first database table to a target file path according to a push mode indicated by the push mode identifier.
Optionally, the parameter generating module is further configured to: and receiving input unloading times, and unloading the database table reaching the unloading time according to the unloading times.
In a third aspect, the present invention also provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable by the processor, wherein the processor executes the computer program to implement the method according to the first aspect and the various alternatives.
In a fourth aspect, the present invention also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the first aspect and various alternatives.
In a fifth aspect, the present invention also provides a computer program product comprising: computer program code which, when run on a computer, causes the computer to perform the steps as described above in the method of the first aspect and in the various alternatives.
Drawings
Fig. 1 is a schematic flowchart of a data offloading method according to an embodiment of the present invention;
FIG. 2 is a schematic structural diagram of a data offloading device according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the drawings of the embodiments of the present invention. In the technical scheme of the invention, the data acquisition, transmission, use and the like meet the requirements of relevant national laws and regulations.
The embodiments of the present invention are directed to dynamic configuration of offloading and pushing multiple data sources, and in order to better understand the scheme in the embodiments of the present invention, some concepts related to the embodiments of the present invention are first described below.
1) A database table is a warehouse that organizes, stores, and manages data according to a data structure, and is an organized, sharable, and uniformly managed collection of large amounts of data stored in a computer. Typically, one or more database tables are located in one database. A database table corresponds to a data source, for example, data representing participation in a campaign, data representing performance of a campaign, data representing a prize delivery, and the like. Database tables from different data sources may belong to one database. Each database table contains a plurality of fields of information (also referred to herein simply as fields), each field representing an attribute of the database table. For example, if a database table includes an unload time field, then the unload time field represents the unload time of the database table.
2) Data unloading is a general term for data processing, for example, data in a database table is extracted according to a rule, and the unloaded data is stored in another form. In the embodiment of the invention, the data unloading is realized by importing the unloading script to generate the unloading file according to the unloading parameters. An unload script may be understood as a tool used to perform an unload process, and an unload parameter is a parameter entered when the unload script is used. Wherein, the uninstalling parameter is generated by field information in the database table. That is to say, the process of data unloading can be understood as generating unloading parameters according to field information in a database table, importing the unloading parameters into an unloading script, generating an unloading file, and running the unloading file to unload data.
3) And the data unloading mode comprises a full unloading mode and an incremental unloading mode. Full offload refers to the unloading of all data in a database table at once. It is understood that all data in a database table is unloaded at the time of unloading, regardless of whether the amount of data in the database table increases or decreases. Incremental unloading refers to unloading the data of a database table within a preset time interval. It will be appreciated that some data may also be entered or deleted by the database table over time, for example, new prize settings may be added over time and new prize data entered. The incremental unloading can realize processing of data in a preset time interval, for example, unloading of newly-entered data in the preset time interval is more flexible.
4) Pushing of data refers to pushing (sending) data to a data analysis system to analyze the data through the data analysis system. For example, data of one or more data sources may be pushed to the data analysis system for different activities, e.g., data representing activity participation is sent to the data analysis system. The data analysis system analyzes the data to obtain the activity participation condition, so that the activity plan is adjusted according to the current activity participation condition.
Statistical analysis is performed on data according to the source of the data, and the data of the same data source is usually stored in a form of a database table. An offload file for offloading database tables may be generated by a database tool, e.g., Oracle, and sent to the data analysis system. In this way, the data analysis system receives the unloading file, and reloads the unloading file to obtain the data in the database table. However, currently, only one database table is unloaded (i.e. processing, such as extracting partial data, etc.) at a time in a scenario with multiple data sources, and the unloading efficiency is low. And since different data sources have different service execution dates, the different data sources are unloaded uniformly, and unnecessary data may be unloaded.
In view of this, a solution of the embodiment of the present invention is provided, in which the uninstalling time of each database table (data of different data sources) may be dynamically configured, so that the uninstalling times of the data of different data sources may be the same, and thus multiple database tables may be uninstalled at a time, thereby improving the uninstalling efficiency. In addition, because the unloading time of each database table is dynamically configured according to the requirements, the condition that a plurality of database tables unloaded simultaneously comprise database tables which do not need to be unloaded is also avoided. In addition, after the database table is unloaded, the unloaded database table can be pushed to a data analysis system, and the pushing time can be dynamically configured, so that the method is more flexible.
To facilitate understanding of the scheme provided by the embodiment of the present invention, a data offloading method provided by the embodiment of the present invention is described below with reference to the accompanying drawings.
Fig. 1 is a schematic flow chart of a data unloading method according to an embodiment of the present invention. The method may be performed by an electronic device having a processing function, such as a mobile phone, a tablet computer (PAD), a Personal Computer (PC), or the like. Of course, the type of the electronic device in the embodiment of the present invention is not limited thereto. In the following description, the method is performed by an electronic device as an example.
S101, the electronic equipment generates a database table list to be unloaded according to the plurality of database tables.
The database tables can represent data from one data source, and the number and the types of the database tables are different according to different data sources. Taking a marketing campaign as an example, the data sources may include data representing participation in the campaign, data representing performance of the campaign, data representing award delivery, and so forth. The data of the various data sources exist in the form of database tables. Each database table may set one or more fields (also referred to as field information) to identify corresponding attribute information. For example, an unload time field may be set for a database table to indicate the time to unload the database table. As another example, an offload mode field may be set for a database table to indicate an offload mode to be employed to offload the database table. Of course, the database table may also include more fields, which are not described in detail here.
According to actual requirements, the data of partial data sources or all data sources may need to be analyzed. Following the above example, the data of the activity participation situation (i.e. a certain database table) may need to be analyzed to obtain the actual activity participation situation, so as to adjust the activity policy according to the actual activity participation situation in the following. Before analyzing the data, the data may be offloaded and the offloaded data may be sent to a data analysis system. And unloading the data, namely extracting the data in the database table according to the rule. For example, there may be multiple database tables that the electronic device may process to generate a list of database tables to be unloaded.
The database table list may include a plurality of database tables that are unloaded, the database table list including a plurality of field information including field information of the plurality of database tables, and more other fields. For example, the database table list further includes an index field and a plurality of field information for the plurality of database tables, wherein the plurality of field information includes a database table name field, an unload mode field, an incremental unload mode field, an unload time field, and the like. The uninstall time field may include a last uninstall time field, a next uninstall time field, an uninstall time interval field, and the like. The last unload time field indicates the time the database table was last unloaded and the next unload time field indicates the time the database table was next unloaded, wherein the time the database table was next unloaded is determined by the time the database table was last unloaded and the time indicated by the unload time interval field. It should be noted that the fields included in the above list of database tables are only examples, and further fields, such as the push information field, may be included as well, as will be described below.
And S102, the electronic equipment generates unloading parameters according to the field information of the database table list.
And after the electronic equipment generates a database table list to be unloaded, unloading according to the field information of the database table list. The electronic equipment can acquire part of field information or all of field information in the database table list to generate unloading parameters, so that one database table or a plurality of database tables in the database table list can be unloaded according to the unloading parameters.
For example, the electronic device may generate an unload parameter characterizing an unload pattern from unload pattern information in a database table list. The offload mode information may indicate a mode of offloading data. In an embodiment of the present invention, the modes of offloading data include a full offload mode and an incremental offload mode. If the unload mode is full unload mode, all data from the start date of the collected data to the unload date is unloaded. It can also be understood that the data in the database table is completely unloaded when the set unloading time is reached. The unloading time of different database tables can be the same, so that the aim of unloading a plurality of database tables at the same time can be fulfilled. Incremental unload mode refers to unloading the data that the database table adds within a preset time interval compared to full unload mode. Different database tables can be set with different preset time intervals, so that the unloading time of each database table is independent.
The preset time interval can be dynamically set according to the unloading requirement, and refers to the preset time interval from the last unloading time. For convenience of description, the unloading time of this time is recorded as the ith unloading time, and the unloading time of the next time is recorded as the (i + 1) th unloading time. The (i + 1) th unloading time is the sum of the (i) th unloading time and a preset time interval. The field information included in the database table list may include unload time information, e.g., the database table list includes an ith unload time field, an i +1 th unload time field, and a time interval field. The time interval field may indicate a preset time interval. In the incremental unload mode, the electronic device may obtain unload time information and generate unload parameters. The unloading parameters include the ith unloading time and time interval. In addition, the electronic device may also determine the unloading parameter of the (i + 1) th time according to the unloading time of the (i + 1) th time and the time interval, for example, the unloading time of the (i + 1) th time.
It should be noted that, in the full offload mode, since there is no relation with the preset time interval, in the full offload mode, the electronic device does not need to determine the offload time according to the time interval field.
S103, the electronic equipment imports the determined uninstalling parameters into an uninstalling script to generate an uninstalling file.
After the electronic device generates the unloading parameters according to the field information, the unloading parameters of one or more database tables in the database table list to be unloaded can be led into the unloading script, and unloading files corresponding to the database tables are generated. The uninstalled script may be a script edited by Shell, a script edited by JAVA, or the like.
Running the offload file may effect an offload of one or more of the database tables in the database table list according to the offload parameters. Take the case that the database table category includes a database table a, a database table B and a database table C, and the unloading parameter includes unloading time. The ith unloading time of the database table A is d1, and the preset time interval is t 1; the ith unloading time of the database table B is d2, and the preset time interval is t 2; the i-th unloading time of the database table C is d3, and the preset time interval is t 3. Wherein d1, d2 and d3 can be the same, so that the database table A, the database table B and the database table C can be unloaded at one time, and the unloading efficiency is obviously improved compared with the condition that only one database table can be unloaded at one time at present. Even though d1, d2 and d3 are different from each other for the incremental unload mode, d1+ t1, d2+ t2 and d3+ t3 may be the same, so that unloading multiple database tables at once may be achieved for the incremental unload mode. In addition, t1, t2 and t3 may be the same or different, and a user may set t1, t2 and t3 according to actual unloading requirements, so that the method is more flexible. Even if the number of the database tables is large, independent and orderly unloading of each database table can be realized by dynamically configuring the ith unloading time and the preset time interval.
The electronic device may run the offload file via the scheduling tool. In the embodiment of the present invention, a plurality of scheduling tools, such as crontab, springbatch, etc., may be provided. And aiming at different application scenes, a matched scheduling tool can be selected and used, and the application range is wide. Specifically, taking crontab as an example, generally, the running time of the crontab tool is in hours, and the running interval of the crontab tool may be referred to as a scheduling interval. For any database table, the ith unloading time can be set by a user through a crontab, and the scheduling interval of the crontab can be set to be different from the preset time interval. For example, the user sets the i-th offload time to d4 and the scheduling interval to 1 hour. The electronic device executes the crontab command to d4 to unload the corresponding database table. After unloading is completed, the unloading time d5 is updated according to the unloading time d4 and the preset time interval. Thus, the electronic device unloads the data between the completion database tables d4 and d 5. After that, the data in the preset time interval from d5 in the database is unloaded in 1 hour interval. Following the example above, assuming that d1 is d2 is d4, then d4 is reached and database table a and database table B are unloaded. Assuming d3 is d4+3 days, then at d4+3, the database table C is unloaded.
After the electronic device runs the uninstalled file, the (i + 1) th uninstalling time can be updated to be the ith uninstalling time, and then a new (i + 1) th uninstalling time is generated according to the ith uninstalling time and a preset time interval. The preset time intervals corresponding to different database tables may be different. Following the example above, the user sets the number of uninstallations to 3 via the crontab, and the scheduling interval is one hour. Taking the database table a as an example, the first unloading time is d1, the preset time interval is 1 day, before the first unloading, the ith unloading time is d1, and the i +1 th unloading time is d1+ 1. When d1 is reached, the first unloading is carried out, after the first unloading, the i-th unloading time is d1+1, and the i + 1-th unloading time is d1+ 2; reaching d1+1, a second unload is performed. After the second unloading, the ith unloading time is d1+2, and the (i + 1) th unloading time is d1+ 3; reaching d1+2, a third unload is performed. After the third unloading, the i-th unloading time is d1+3, and the i + 1-th unloading time is d1+ 4.
Therefore, the embodiment of the invention can dynamically configure the unloading parameters of the unloading database table, such as unloading time, unloading interval time and the like, thereby realizing the unloading of a plurality of database tables at one time. In addition, the unloading times can be dynamically configured, and the database tables which are not unloaded actually in the unloading period can be continuously unloaded, so that the database tables are prevented from being unloaded.
After the database table is unloaded, the electronic device can also push the database table, that is, the database table is sent to the data analysis system, and the data analysis is realized through the data analysis system. The database tables pushed to the data analysis system are usually the database tables after unloading, so the electronic device can determine the database tables that have been unloaded before pushing the database tables.
In a possible implementation manner, the database table list sets an uninstalling identifier to indicate whether uninstalling of the corresponding database table is completed, for example, the uninstalling identifier is "0" to indicate that uninstalling is completed, the uninstalling identifier is "1" to indicate that uninstalling is not completed. It should be understood that for any database table, the electronic device may reset the unload flag after unloading the database table.
Before pushing the database table, the electronic device may acquire field information representing the uninstallation identifier in the database table, and further determine the database table in which the uninstallation is completed. Take the first database table as an example of the database whose unloading is completed. The user can obtain the push information of the first database table and push the first database table according to the push information. The push information includes a storage path and a target file path, the storage path refers to a path storing the first database table, and the target file path refers to a path to which the first database table is to be pushed, for example, a path storing the first database table in the data analysis system. The storage path may be an uninstall file corresponding to the database table in which the uninstall is completed. The electronic device pushes the first database table, which can also be understood as sending the uninstalled file corresponding to the first database table to the target file path.
The embodiment of the invention provides a plurality of modes for pushing the database table, and can also be understood as a mode for pushing the database table, which is referred to as a pushing mode for short. The push mode includes, for example, a mode of pushing a database table in a jfserver, ftp, httpclient, or the like. Aiming at the interface of the data analysis system, the electronic equipment can select a matched push mode, and the application range is wider. The predefined push modes include a jfserver push mode, an ftp push mode and an httpclient push mode. The push patterns for different database table configurations may be the same or different. Before the electronic equipment pushes the first database table, the electronic equipment can acquire the push mode identifier corresponding to the first database table, so that the first database table is pushed according to the push mode indicated by the push mode identifier. Taking the push mode identifier indicating the ftp push mode as an example, the electronic device obtains the uninstalled file of the database table a according to the storage path, and then sends the uninstalled file to the target file path in the ftp push mode.
Similar to the unload time, the user may also set the push time for the database table. The push time may be different for different database tables. The user can also set the pushing interval of the database table, and the pushing is not immediately carried out after the unloading is finished, but carried out after the pushing interval. The different database table push intervals may also be different. Similarly, the user can set the pushing times of the database table so as to avoid missing the pushing of the database table as much as possible.
The following describes the apparatus provided by the embodiment of the present invention with reference to the drawings.
Referring to fig. 2, based on the same inventive concept, an embodiment of the invention provides a data offloading device 200, where the data offloading device 200 can implement the process shown in fig. 1. The data offloading device 200 includes a list generation module 201, a parameter generation module 202, and an offloading module 203. The list generating module 201 is configured to generate a database list to be unloaded according to a plurality of database tables, where the database list includes respective indexes of the plurality of database tables and a plurality of field information, and the plurality of field information includes field information corresponding to each of the plurality of database tables. The parameter generating module 202 is configured to generate an unloading parameter according to the field information of the database table list, where the unloading parameter is used to unload the plurality of database tables. The uninstalling module 203 is configured to import the uninstalling parameters into an uninstalling script to generate an uninstalling file, where the uninstalling script is used to uninstall the plurality of database tables.
Optionally, the parameter generating module 202 generates an unloading parameter according to field information of the database table list, and is specifically configured to: and acquiring the unloading mode information, and generating the unloading parameters according to the unloading mode information.
Optionally, the parameter generating module 202 generates an unloading parameter according to the field information of the database table list, and is further configured to: acquiring unloading time information, and generating unloading parameters according to the unloading time information, wherein the unloading parameters comprise ith unloading time and a preset time interval, and the preset time interval and the ith unloading time are used for determining the (i + 1) th unloading time.
Optionally, after the uninstalling module 203 imports the uninstalling parameters into the uninstalling script to generate the uninstalling script, the uninstalling module is further configured to: and updating the (i + 1) th unloading time to the (i) th unloading time, and then acquiring the (i) th unloading time and a preset time interval to generate new (i + 1) th unloading time. Wherein, the preset time intervals corresponding to different database tables are different.
Optionally, the data offloading device 200 is further configured to: and acquiring an unloading identifier corresponding to the first database table, and acquiring push information if the unloading identifier indicates that the unloading of the first database table is finished. It should be noted that the push information includes a storage path of the first database table and a target file path. And then, the electronic equipment acquires the first database table from the storage path and pushes the first database table to the target file path.
Optionally, the data offloading device 200 is further configured to: and acquiring a pushing mode identifier, and pushing the first database table to a target file path according to a pushing mode indicated by the pushing mode identifier.
Optionally, the data offloading device 200 is further configured to: and receiving input unloading times, and unloading the database table reaching the unloading time according to the unloading times.
Referring to fig. 3, based on the same inventive concept, an embodiment of the invention provides an electronic device 300, where the electronic device 300 can implement the process shown in fig. 1. The electronic device 300 comprises at least one processor 301 and a memory 302 communicatively connected to the at least one processor 301, wherein the processor 301 implements any of the above-mentioned data offloading methods when executing a computer program.
The at least one processor 301 may be configured to generate a database table list to be unloaded according to a plurality of database tables, where the database table list includes respective indexes and a plurality of field information of the plurality of database tables, and the plurality of field information includes field information corresponding to each of the plurality of database tables. Generating unloading parameters according to field information of the database table list, wherein the unloading parameters are used for unloading a plurality of database tables; and importing the unloading parameters into an unloading script to generate an unloading file, wherein the unloading script is used for unloading the plurality of database tables.
The electronic device 300 may be configured to perform the method provided by the embodiment shown in fig. 1, for example, the electronic device described above. Therefore, regarding the functions that can be realized by the functional modules in the electronic device 300, reference may be made to the corresponding description in the embodiment shown in fig. 1, which is not repeated herein.
Embodiments of the present invention also provide a computer storage medium, where the computer storage medium stores computer instructions, and when the computer instructions are executed on a computer, the computer is caused to execute the method as described in fig. 1.
An embodiment of the present invention further provides a computer program product, where the computer program product includes: computer program code which, when run on a computer, causes the computer to perform the method as described in figure 1.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations as well.
Claims (14)
1. A method of data offloading, comprising:
generating a database table list to be unloaded according to a plurality of database tables, wherein the database table list comprises respective indexes and a plurality of field information of the plurality of database tables, and the plurality of field information comprises the field information corresponding to each database table in the plurality of database tables;
generating unloading parameters according to the field information of the database table list, wherein the unloading parameters are used for unloading the plurality of database tables;
and importing the unloading parameters into an unloading script to generate an unloading file, wherein the unloading script is used for unloading the plurality of database tables.
2. The method of claim 1, wherein the plurality of field information of the database table list includes unload mode information, the unload mode information including an unload mode, the unload mode including a full unload mode or an incremental unload mode, the full unload mode indicating to unload data in all of the database tables included in the database table list, the incremental unload mode indicating to unload data in the plurality of database tables added within a preset time interval; generating unloading parameters according to the field information of the database table list, wherein the unloading parameters comprise:
and acquiring the unloading mode information, and generating the unloading parameters according to the unloading mode information.
3. The method of claim 2 wherein the plurality of field information of the database table list further includes unload time information indicating an unload time of the database table in the incremental unload mode, generating unload parameters from the field information of the database table list including:
and acquiring the unloading time information, and generating the unloading parameters according to the unloading time information, wherein the unloading parameters comprise ith unloading time and a preset time interval, and the preset time interval and the ith unloading time are used for determining the (i + 1) th unloading time.
4. The method of claim 3, wherein after importing the uninstall parameters into an uninstall script to generate an uninstall file, the method further comprises:
updating the i +1 th unloading time to the i-th unloading time;
and acquiring the ith unloading time and a preset time interval, and generating new (i + 1) th unloading time, wherein the preset time intervals corresponding to different database tables are different.
5. The method of any one of claims 1-4, wherein the plurality of fields of the database table list include an unload flag, the unload flag to indicate whether the unloading of the database table is complete, the method further comprising:
acquiring the unloading identifier corresponding to a first database table, and acquiring push information if the unloading identifier indicates that the unloading of the first database table is finished, wherein the push information comprises a storage path and a target file path of the first database table;
and acquiring the first database table from the storage path, and pushing the first database table to the target file path.
6. The method of claim 5, wherein the push information further includes a push mode identification, the push mode identification indicating a push mode, pushing the first database table to the target file path, comprising:
acquiring the pushing mode identification;
and pushing the first database table to the target file path according to the pushing mode indicated by the pushing mode identifier.
7. The method of claim 3 or 4, further comprising:
receiving input uninstalling times;
and unloading the database table reaching the unloading time according to the unloading times.
8. A data offloading device, comprising:
the database table list comprises indexes and field information of the database tables, and the field information comprises the field information corresponding to each database table in the database tables;
the parameter generating module is used for generating unloading parameters according to the field information of the database table list, and the unloading parameters are used for unloading the plurality of database tables;
and the unloading module is used for importing the unloading parameters into an unloading script to generate an unloading file, and the unloading script is used for unloading the plurality of database tables.
9. The apparatus of claim 8, wherein the plurality of field information of the database table list includes offload mode information, the offload mode information including an offload mode, the offload mode including a full offload mode or an incremental offload mode, the full offload mode indicating to offload data in all database tables included in the database table list, the incremental offload mode indicating to offload data in the plurality of database tables that is increasing within a preset time interval, the parameter generation module specifically to:
and acquiring the unloading mode information, and generating the unloading parameters according to the unloading mode information.
10. The apparatus of claim 9, wherein the plurality of field information of the database table list further includes unload time information indicating an unload time of the database table in the incremental unload mode, the parameter generation module further to:
and acquiring the unloading time information, and generating the unloading parameters according to the unloading time information, wherein the unloading parameters comprise ith unloading time and a preset time interval, and the preset time interval and the ith unloading time are used for determining the (i + 1) th unloading time.
11. The apparatus of claim 10, wherein the offload module is specifically configured to:
updating the i +1 th unloading time to the i-th unloading time;
and acquiring the ith unloading time and a preset time interval, and generating new (i + 1) th unloading time, wherein the preset time intervals corresponding to different database tables are different.
12. An electronic device, comprising:
a memory for storing program instructions;
a processor for calling program instructions stored in said memory and for executing the method according to any one of claims 1-7 in accordance with the obtained program instructions.
13. A computer-readable storage medium, characterized in that it stores a computer program comprising program instructions which, when executed by a computer, cause the computer to carry out the steps of the method according to any one of claims 1 to 7.
14. A computer program product, the computer program product comprising: computer program code for causing a computer to carry out the steps of the method as claimed in any one of claims 1 to 7 when said computer program code is run on a computer.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210706934.1A CN114925074A (en) | 2022-06-21 | 2022-06-21 | Data unloading method, device, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210706934.1A CN114925074A (en) | 2022-06-21 | 2022-06-21 | Data unloading method, device, equipment and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114925074A true CN114925074A (en) | 2022-08-19 |
Family
ID=82813754
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210706934.1A Pending CN114925074A (en) | 2022-06-21 | 2022-06-21 | Data unloading method, device, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114925074A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030177146A1 (en) * | 2002-03-18 | 2003-09-18 | International Business Machines Corporation | Method, system, and program product for migrating data from one data base management system to another data base management system |
| US20050114404A1 (en) * | 2003-11-21 | 2005-05-26 | Bmc Software, Inc. | Database table version upload |
| CN102375891A (en) * | 2011-11-15 | 2012-03-14 | 山东浪潮金融信息系统有限公司 | Implementation tool for unloading and loading incremental data |
| CN109508355A (en) * | 2018-10-19 | 2019-03-22 | 平安科技（深圳）有限公司 | A kind of data pick-up method, system and terminal device |
| CN112687364A (en) * | 2020-12-24 | 2021-04-20 | 宁波金唐软件有限公司 | Hbase-based medical data management method and system |
| CN113641742A (en) * | 2021-08-05 | 2021-11-12 | 广东电网有限责任公司 | Data extraction method, device, equipment and storage medium |
-
2022
- 2022-06-21 CN CN202210706934.1A patent/CN114925074A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030177146A1 (en) * | 2002-03-18 | 2003-09-18 | International Business Machines Corporation | Method, system, and program product for migrating data from one data base management system to another data base management system |
| US20050114404A1 (en) * | 2003-11-21 | 2005-05-26 | Bmc Software, Inc. | Database table version upload |
| CN102375891A (en) * | 2011-11-15 | 2012-03-14 | 山东浪潮金融信息系统有限公司 | Implementation tool for unloading and loading incremental data |
| CN109508355A (en) * | 2018-10-19 | 2019-03-22 | 平安科技（深圳）有限公司 | A kind of data pick-up method, system and terminal device |
| CN112687364A (en) * | 2020-12-24 | 2021-04-20 | 宁波金唐软件有限公司 | Hbase-based medical data management method and system |
| CN113641742A (en) * | 2021-08-05 | 2021-11-12 | 广东电网有限责任公司 | Data extraction method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110245149B (en) | Metadata version management method and device | |
| CN110737594B (en) | Database standard conformance testing method and device for automatically generating test cases | |
| CN105138371A (en) | Software upgrading method and device | |
| CN112631686A (en) | Data processing method, data processing device, computer equipment and storage medium | |
| CN110865828A (en) | Data upgrading method, device, equipment and computer readable storage medium | |
| CN113901021A (en) | Method and device for generating upgrading script for multi-version database and electronic equipment | |
| CN110362540B (en) | Data storage and visitor number acquisition method and device | |
| CN113721950A (en) | Resource loss prevention and control processing method and device | |
| CN107688586B (en) | Client data shielding processing method and device | |
| CN110990381B (en) | Processing method and device of server, storage medium and electronic device | |
| CN114925074A (en) | Data unloading method, device, equipment and medium | |
| CN113535206B (en) | Multi-version code upgrading method and system | |
| CN114466387A (en) | Method and device for updating configuration file of base station, storage medium and electronic device | |
| CN115878443A (en) | Interface test script generation method and device, storage medium and electronic device | |
| CN112732242A (en) | Wide table processing script generation method and device | |
| CN113377604A (en) | Data processing method, device, equipment and storage medium | |
| CN111339422A (en) | Recommendation system task management platform, recommendation method and system | |
| CN113760489B (en) | Resource allocation method and device | |
| CN110990475B (en) | Batch task inserting method and device, computer equipment and storage medium | |
| CN114785847B (en) | Network control software development configuration method, terminal and storage medium | |
| CN115941834B (en) | Automatic operation method, device, equipment and storage medium of smart phone | |
| CN116150006A (en) | Test case management method, computer equipment and computer storage medium | |
| CN113703729A (en) | Method, device and equipment for generating program call chain | |
| CN115529237A (en) | Configuration information loading method, device and equipment | |
| CN113076098A (en) | General ledger data processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |