CN104376082A - Method for importing data in data source file to database - Google Patents
Method for importing data in data source file to database Download PDFInfo
- Publication number
- CN104376082A CN104376082A CN201410658208.2A CN201410658208A CN104376082A CN 104376082 A CN104376082 A CN 104376082A CN 201410658208 A CN201410658208 A CN 201410658208A CN 104376082 A CN104376082 A CN 104376082A
- Authority
- CN
- China
- Prior art keywords
- data
- queue
- database
- thread
- data processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method for importing data in a data source file to a database. The method includes the step A of reading the data in the data source file, the step B of storing the data into an original data queue line by line, the step C of setting multiple data processing storage threads, and the step D of preprocessing the data in the original data queue through the data processing storage threads and writing the preprocessed data into the database in parallel through the data processing storage threads. Parallel importing of the data is achieved through the threads, hardware resources are fully utilized, and therefore storage efficiency is improved and the data importing speed is greatly increased.
Description
Technical field
The present invention relates to field of data storage, particularly, the present invention relates to a kind of the data importing in data source file to the method in database.
Background technology
Along with the continuous growth of application system quantity, a large amount of data importing demands is thereupon raw, and existing database (such as, INFORMIX, MYSQL etc.) only provides the derivative instrument of simple rule in order, not fully up to expectations in its efficiency and extendability.
The database that current Ge Jia bank uses is mostly INFORMIX database, and the frequently-used data introduction method that its official provides has load and dbload two kinds, but these two kinds of methods all also exist significant limitation in data processing:
First, the boot speed of these two kinds of methods is slower.For the 1000000 row often data file that arranges of row 27: when this data file being imported in database by load mode, reporting an error in any importing process all can cause that factor data amount is excessive causes Database lock resource exhaustion and rollback.When this data file being imported in database by dbload mode, it needs 184440 milliseconds consuming time.
Secondly, these two kinds of methods can only be implemented linearly to read importing to text.Load and dbload is owing to being the instrument that official provides, and therefore not easily extensible, can only carry out data importing by the order linear of often going.
3rd, the wrong output format of these two kinds of methods not can customize.The error message of load method is output on screen, and once make mistakes, data are all considered as makeing mistakes by the gross, and program no longer continues to process data importing.Although dbload can self-defined error log name, form is fixing, cannot realize self-defined.
Based on above reason, be badly in need of a kind of higher storage efficiency that has, and above-mentioned defect can be overcome, data lead-in method, thus meet the demand that big data quantity imports.
Summary of the invention
Along with the enterprise application system of large data age, Data Migration or process become frequent all the more, and traditional data importing mode can not meet existing demand.The importing of millions data volume needs cost several hours easily, and therefore the importing efficiency of data is in urgent need to be improved.Given this, the invention provides a kind of the data importing in data source file to the method in database.
The invention discloses a kind of the data importing in data source file to the method in database, comprise the following steps: A. reads the data in described data source file; B. described data line by line stored in in raw data queue; C. multiple data processing warehouse-in thread is set; D. by described multiple data processing warehouse-in thread, pre-service is carried out to the data in described raw data queue, and by described multiple data processing warehouse-in thread, described pretreated data parallel is written in described database.
Especially, described step D specifically comprises: a. data processing warehouse-in thread described in each reads the data of predetermined quantity respectively from described raw data queue, and carries out pre-service one by one; B. data processing warehouse-in thread described in each respectively described pretreated data stored in respective pre-service queue; C. described in each, data processing warehouse-in thread is written to the data parallel in described pre-service queue in described database; D. described in each, data processing warehouse-in thread empties the data in described pre-service queue; E. described in each, data processing warehouse-in thread confirms whether also have untreated data in described raw data queue respectively: if there are untreated data, then perform step a.
Especially, specifically comprise at described step c: I. extracts described data successively from described pre-service queue; Whether the data be extracted described in II. confirming exist mistake; When there is an error, implementation step:
Vi. the position of described misdata in described pre-service queue is determined;
Vii. the data before described position are written in described database;
Viii. the object information of the misdata on described position is written to error queue
In;
Ix. to delete in described pre-service queue on described position and before data;
X. step I is performed;
When there is not mistake: the described data extracted are written in described database.
Especially, described object information comprises: positional information, content, error reason.
Especially, also step is comprised: process described error queue by error handling processing thread.
Especially, the information in error queue according to user-defined formatted output.
Especially, the degree of depth of described raw data queue and the degree of depth of described pre-service queue are configurableization.
Especially, in step a, the degree of depth based on described pre-service queue determines described predetermined quantity.
Especially, the degree of depth of described pre-service queue is 50.
Especially, described database is the database meeting JDBC specification.
Especially, described database is INFORMIX database or MYSQL database.
In sum, method tool disclosed according to the present invention has the following advantages:
1) the present invention realizes data parallel importing by multithreading, takes full advantage of hardware resource, thus improves storage efficiency, the speed of data importing is improved greatly.For 1,000,000 row often row 27 arrange data file, complete data importing by the method disclosed in the present and only need 59473 milliseconds;
2) the present invention is applicable to all databases meeting JDBC specification, thus realizes the unified data importing process of integration across database;
3) the present invention can process every data line by demand personalization, breaks away from the not extendible drawback of reading by column that database carries instrument; And the present invention can customize Output rusults, such as direct output screen, preservation file, preservation database.
Accompanying drawing explanation
By reading the detailed description done non-limiting example done with reference to the following drawings, other features, objects and advantages of the present invention will become more apparent:
Fig. 1 show disclosed according to the present invention the data importing in data source file to the method flow diagram in database;
Fig. 2 show disclosed according to the present invention the data importing in data source file to the process schematic in database;
Fig. 3 shows data processing threads disclosed according to the present invention and pretreated data is written to schematic diagram in database; And
Fig. 4 shows the method flow diagram that process data importing disclosed according to the present invention makes mistakes.
Embodiment
In the specific descriptions of following preferred embodiment, with reference to the accompanying drawing formed appended by a part of the present invention.Appended accompanying drawing shows by way of example and can realize specific embodiment of the present invention.The embodiment of example is not intended to limit according to all embodiments of the present invention.Be appreciated that under the prerequisite not departing from scope of the present invention, other embodiments can be utilized, also can carry out amendment that is structural or logicality.Therefore, following specific descriptions are also nonrestrictive, and scope of the present invention limited by appended claim.
Below with reference to accompanying drawing 1-4, illustrate a kind of the data importing in data source file to the method in database.It should be noted that, although describe the step of method in accompanying drawing with particular order, but this not requires or hint must perform these operations according to this particular order, or the operation shown in must performing all could realize the result of expectation, on the contrary, the step described in process flow diagram can change execution sequence.Additionally or alternatively, some step can be omitted, multiple step be merged into a step and perform, and/or a step is decomposed into multiple step and perform.
As shown in Figure 1, 2, in step 110, the file in file read module reads thread 210 and from data source file 200, reads data by the mode read line by line.Wherein, because file reading speed is in this step far longer than the speed of data processing and storage in subsequent step, therefore configured to use only a file in this programme and read thread in order to read the data in data source file 200.
In the step 120, file reading thread 210 is written to the data read from data source file 200 in the raw data queue 220 simultaneously started.The degree of depth of this queue 220 can by User Defined.In a preferred embodiment, the degree of depth of this raw data queue 220 can be decide according to the hardware configuration of data source file and gatherer.When internal memory that is comparatively large when data source file and/or gatherer is larger, it is deep that the degree of depth of this raw data queue 220 can set.In a specific embodiment, the degree of depth of this raw data queue 220 is 5000, that is: this raw data queue can preserve 5000 row data.
After the data in raw data queue 220 are read by follow-up thread (data processing warehouse-in thread 230), these data are by deleted, and file reads thread 210 to be put into raw data queue 220 by obtaining new data from data source file 200, and this process will be implemented repeatedly until the data of data source file 200 are all read complete.
In step 130, enter in library module to set multiple data processing warehouse-in thread 230 at batch, the function of the plurality of data processing warehouse-in thread 230 is all identical.This data processing warehouse-in thread 230 for reading data from raw data queue 220, and is written to data in database 240 after the pre-treatment, and this process will repeatedly be implemented until data are all imported into complete in raw data queue 220.In the present invention, this database 240 is the databases meeting JDBC specification, such as: INFORMIX database or MYSQL database.
See Fig. 3, in each data processing warehouse-in thread 230, be respectively equipped with a pre-service queue 330.Because the processing speed of data processing warehouse-in thread 230 is comparatively slow, the degree of depth of therefore pre-service queue 330 is much smaller than the degree of depth of raw data queue 220.
In a preferred embodiment, the degree of depth of pre-service queue 330 and/or the quantity of data processing warehouse-in thread 230 can be configured according to system performance.Such as, the data processing speed that the quantity of the degree of depth of pre-service queue 330 and/or data processing warehouse-in thread 230 can put thread 230 in storage according to data processing sets.When the data processing speed of data processing warehouse-in thread 230 is very fast, the degree of depth of pre-service queue 330 is comparatively dark, the negligible amounts of data processing warehouse-in thread 230; When the data processing speed of data processing warehouse-in thread 230 is slower, the degree of depth of pre-service queue 330 is more shallow, and the quantity of data processing warehouse-in thread 230 is more.
It is to be noted not have specific sequencing between step 110 and step 130, both can implement simultaneously, also can successively implement.
In step 140, each data processing warehouse-in thread 230 reads the data of predetermined quantity respectively from raw data queue 220, and carries out pre-service one by one.Wherein, described predetermined quantity can be determined according to the degree of depth of pre-service queue 330, and described predetermined quantity equals the degree of depth of pre-service queue 330 in the present embodiment.Described pre-service comprises according to pre-service interface the form required for the data transformations one-tenth read from raw data queue 220.In the present embodiment, described pre-service can be assembled into parameter list in JDBC required for PreparedStatement data.Pretreated data will be easy to be directed in JDBC database more.
In step 150, each data processing warehouse-in thread 230 respectively pretreated data stored in respective pre-service queue 330.And after this pre-service queue 330 has been expired, in a step 160, data processing warehouse-in thread 230 can be written to the data in described pre-service queue in described database 240.Wherein, be concurrently pretreated data are written in database 240 between multiple data processing warehouse-in thread 230, the mode be written in parallel to by this, the time of data processing and write obtains compression, thus improves the efficiency of data importing.
In step 170, after the data importing in respective pre-service queue 330 to database 240, data processing warehouse-in thread 230 empties the data in this pre-service queue 330 respectively.And in step 180, data processing warehouse-in thread 230 confirms whether also have untreated data in raw data queue 220: if there are untreated data, then repeated execution of steps 140 to 180, until the data in raw data queue 220 are all directed in database 240.
In a step 160, when each data processing warehouse-in thread 230 writes data in database 240, occur sometimes to write the situation of makeing mistakes.For this situation, Fig. 4 shows the method flow diagram that process data importing disclosed according to the present invention makes mistakes, and described step 160 specifically comprises the following steps:
In the step 161, each data processing warehouse-in thread 230 can extract pre-service and need to be written into the data in database 240 from respective pre-service queue 330.
Confirm whether these data exist mistake in step 162.
Following steps are implemented: in step 163, confirm the position of this mistake in pre-service queue 330 when the data finding that this is extracted exist mistake; In step 164, the data of not makeing mistakes before this position are written in database 240; In step 165, the object information of this error data is written in error queue 340, wherein, this object information comprises: positional information (such as: the line number of these data in data source file 200, or the location number of this misdata in pre-service queue 330), content, any information relevant to this misdata such as reason of makeing mistakes; In step 166, recorded data dump, that is: in pre-service queue 330 on deletion error position and before data.Finally be back in step 161, to continue to extract remaining data.
In a specific embodiment, this error queue 340 is processed by follow-up error handling processing thread, this processing mode comprises: described object information is written in file, database or the JMS specified, or the information in error queue according to user-defined formatted output.
When finding that these data be extracted do not exist mistake, then implementation step 167: the data extracted are written in database 240.
Finally, the described result of data write and the result of error message can be exported by self-defining mode, such as direct output screen, preservation file, preservation database.
Illustrate that data processing warehouse-in thread is to the process of misdata below by way of a concrete example: such as, be in the pre-service queue of 50 a degree of depth, there is mistake in the data on the 20th position in queue, then data processing warehouse-in thread can extract data successively from pre-service queue, when extracting the 20th data, data processing warehouse-in thread finds that these data exist mistake, so this thread can stop continuing to extract data, but the data on 1-19 position are before written in database, and the object information of the data on the 20th position is written to error queue, then the data on 1-20 position are deleted.Data processing warehouse-in thread continues to extract data from the 21st position afterwards, until all data are extracted complete and are written in database.
To those skilled in the art, obviously the invention is not restricted to the details of above-mentioned one exemplary embodiment, and when not deviating from spirit of the present invention or essential characteristic, the present invention can be realized in other specific forms.Therefore, in any case, all should embodiment be regarded as exemplary, and be nonrestrictive.In addition, significantly, " comprising " one word do not get rid of other elements and step, and wording " one " does not get rid of plural number.Multiple elements of stating in device claim also can be realized by an element.First, second word such as grade is used for representing title, and does not represent any specific order.
Claims (11)
1. the data importing in data source file to the method in database, comprise the following steps:
A. the data in described data source file are read;
B. described data line by line stored in in raw data queue;
C. multiple data processing warehouse-in thread is set;
D. by described multiple data processing warehouse-in thread, pre-service is carried out to the data in described raw data queue, and by described multiple data processing warehouse-in thread, described pretreated data parallel is written in described database.
2. method according to claim 1, wherein, described step D specifically comprises:
A. described in each, data processing warehouse-in thread reads the data of predetermined quantity respectively from described raw data queue, and carries out pre-service one by one;
B. data processing warehouse-in thread described in each respectively described pretreated data stored in respective pre-service queue;
C. described in each, data processing warehouse-in thread is written to the data parallel in described pre-service queue in described database;
D. described in each, data processing warehouse-in thread empties the data in described pre-service queue;
E. described in each, data processing warehouse-in thread confirms whether also have untreated data in described raw data queue respectively: if there are untreated data, then perform step a.
3. method according to claim 2, wherein, specifically comprises at described step c:
I. from described pre-service queue, described data are extracted successively;
Whether the data be extracted described in II. confirming exist mistake;
When there is an error, implementation step:
I. the position of described misdata in described pre-service queue is determined;
Ii. the data before described position are written in described database;
Iii. the object information of the misdata on described position is written in error queue;
Iv. to delete in described pre-service queue on described position and before data;
V. step I is performed;
When there is not mistake: the described data extracted are written in described database.
4. method according to claim 3, wherein, described object information comprises: positional information, content, error reason.
5. method according to claim 3, wherein, also comprises step: process described error queue by error handling processing thread.
6. method according to claim 3, wherein, the information in error queue according to user-defined formatted output.
7. method according to claim 3, wherein, the degree of depth of described raw data queue and the degree of depth of described pre-service queue are configurableization.
8. method according to claim 7, wherein, in step a, the degree of depth based on described pre-service queue determines described predetermined quantity.
9. method according to claim 7, wherein, the degree of depth of described pre-service queue is 50.
10. method according to claim 3, is characterized in that, described database is the database meeting JDBC specification.
11. methods according to claim 10, is characterized in that, described database is INFORMIX database or MYSQL database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410658208.2A CN104376082B (en) | 2014-11-18 | 2014-11-18 | A method of the data in data source file are imported into database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410658208.2A CN104376082B (en) | 2014-11-18 | 2014-11-18 | A method of the data in data source file are imported into database |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104376082A true CN104376082A (en) | 2015-02-25 |
| CN104376082B CN104376082B (en) | 2019-06-18 |
Family
ID=52554989
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410658208.2A Active CN104376082B (en) | 2014-11-18 | 2014-11-18 | A method of the data in data source file are imported into database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104376082B (en) |
Cited By (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105045917A (en) * | 2015-08-20 | 2015-11-11 | 北京百度网讯科技有限公司 | Example-based distributed data recovery method and device |
| CN105279261A (en) * | 2015-10-23 | 2016-01-27 | 北京京东尚科信息技术有限公司 | Dynamic extensible database filing method and system |
| CN105468705A (en) * | 2015-11-18 | 2016-04-06 | 广东南方通信建设有限公司 | Mobile communication background data file importing method |
| CN106354788A (en) * | 2016-08-23 | 2017-01-25 | 无锡天脉聚源传媒科技有限公司 | File reading method and device |
| WO2017028690A1 (en) * | 2015-08-14 | 2017-02-23 | 阿里巴巴集团控股有限公司 | File processing method and system based on etl |
| CN106599222A (en) * | 2016-12-19 | 2017-04-26 | 广州四三九九信息科技有限公司 | Method and equipment for log streaming type parallel processing |
| CN106909554A (en) * | 2015-12-22 | 2017-06-30 | 亿阳信通股份有限公司 | A kind of loading method and device of database text table data |
| CN107357868A (en) * | 2017-07-03 | 2017-11-17 | 华通信安(北京)科技发展有限公司 | A kind of fast conversion method and device of matlab data formats |
| CN107665195A (en) * | 2016-07-27 | 2018-02-06 | 北京京东尚科信息技术有限公司 | Import the method, apparatus and system of mass data |
| CN107862095A (en) * | 2017-12-07 | 2018-03-30 | 中国银行股份有限公司 | A kind of data processing method and device |
| CN108228730A (en) * | 2017-12-11 | 2018-06-29 | 深圳市买买提信息科技有限公司 | Data lead-in method, device, computer equipment and readable storage medium storing program for executing |
| CN108959292A (en) * | 2017-05-19 | 2018-12-07 | 北京京东尚科信息技术有限公司 | A kind of data uploading method, system and computer readable storage medium |
| CN109033184A (en) * | 2018-06-27 | 2018-12-18 | 中国建设银行股份有限公司 | Data processing method and device |
| CN109857832A (en) * | 2019-01-03 | 2019-06-07 | 中国银行股份有限公司 | A kind of preprocess method and device of payment data |
| CN110362617A (en) * | 2019-06-24 | 2019-10-22 | 北京人大金仓信息技术股份有限公司 | Batch data method and system is quickly exported from database based on more concurrent technologies |
| CN112328542A (en) * | 2020-11-25 | 2021-02-05 | 天津凯发电气股份有限公司 | Method for importing data in heterogeneous data file into database |
| WO2022267837A1 (en) * | 2021-06-23 | 2022-12-29 | 比亚迪股份有限公司 | Data archiving method, electronic device, and readable storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6477535B1 (en) * | 1998-11-25 | 2002-11-05 | Computer Associates Think Inc. | Method and apparatus for concurrent DBMS table operations |
| CN101515291A (en) * | 2009-03-26 | 2009-08-26 | 北京泰合佳通信息技术有限公司 | Method for leading data into database in a batch way and system thereof |
| CN103049533A (en) * | 2012-12-23 | 2013-04-17 | 北京人大金仓信息技术股份有限公司 | Method for quickly loading data into database |
| CN103092840A (en) * | 2011-10-28 | 2013-05-08 | 上海邮电设计咨询研究院有限公司 | Method for acquiring self-increment mass data files from multiple sources |
-
2014
- 2014-11-18 CN CN201410658208.2A patent/CN104376082B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6477535B1 (en) * | 1998-11-25 | 2002-11-05 | Computer Associates Think Inc. | Method and apparatus for concurrent DBMS table operations |
| CN101515291A (en) * | 2009-03-26 | 2009-08-26 | 北京泰合佳通信息技术有限公司 | Method for leading data into database in a batch way and system thereof |
| CN103092840A (en) * | 2011-10-28 | 2013-05-08 | 上海邮电设计咨询研究院有限公司 | Method for acquiring self-increment mass data files from multiple sources |
| CN103049533A (en) * | 2012-12-23 | 2013-04-17 | 北京人大金仓信息技术股份有限公司 | Method for quickly loading data into database |
Non-Patent Citations (1)
| Title |
|---|
| LIANJIE1229: ""Oracle中如何定位报错的行"", 《LINUX公社》 * |
Cited By (24)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017028690A1 (en) * | 2015-08-14 | 2017-02-23 | 阿里巴巴集团控股有限公司 | File processing method and system based on etl |
| CN105045917B (en) * | 2015-08-20 | 2019-06-18 | 北京百度网讯科技有限公司 | A kind of the distributed data restoration methods and device of Case-based Reasoning |
| WO2017028394A1 (en) * | 2015-08-20 | 2017-02-23 | 北京百度网讯科技有限公司 | Example-based distributed data recovery method and apparatus |
| CN105045917A (en) * | 2015-08-20 | 2015-11-11 | 北京百度网讯科技有限公司 | Example-based distributed data recovery method and device |
| US10783163B2 (en) | 2015-08-20 | 2020-09-22 | Beijing Baidu Netcom Science And Technology Co., Ltd. | Instance-based distributed data recovery method and apparatus |
| CN105279261A (en) * | 2015-10-23 | 2016-01-27 | 北京京东尚科信息技术有限公司 | Dynamic extensible database filing method and system |
| CN105279261B (en) * | 2015-10-23 | 2019-07-02 | 北京京东尚科信息技术有限公司 | Dynamic scalable database filing method and system |
| CN105468705A (en) * | 2015-11-18 | 2016-04-06 | 广东南方通信建设有限公司 | Mobile communication background data file importing method |
| CN106909554A (en) * | 2015-12-22 | 2017-06-30 | 亿阳信通股份有限公司 | A kind of loading method and device of database text table data |
| CN106909554B (en) * | 2015-12-22 | 2020-08-04 | 亿阳信通股份有限公司 | Method and device for loading database text table data |
| CN107665195A (en) * | 2016-07-27 | 2018-02-06 | 北京京东尚科信息技术有限公司 | Import the method, apparatus and system of mass data |
| CN106354788A (en) * | 2016-08-23 | 2017-01-25 | 无锡天脉聚源传媒科技有限公司 | File reading method and device |
| CN106599222A (en) * | 2016-12-19 | 2017-04-26 | 广州四三九九信息科技有限公司 | Method and equipment for log streaming type parallel processing |
| CN108959292A (en) * | 2017-05-19 | 2018-12-07 | 北京京东尚科信息技术有限公司 | A kind of data uploading method, system and computer readable storage medium |
| CN108959292B (en) * | 2017-05-19 | 2021-03-30 | 北京京东尚科信息技术有限公司 | Data uploading method, system and computer readable storage medium |
| CN107357868A (en) * | 2017-07-03 | 2017-11-17 | 华通信安(北京)科技发展有限公司 | A kind of fast conversion method and device of matlab data formats |
| CN107862095A (en) * | 2017-12-07 | 2018-03-30 | 中国银行股份有限公司 | A kind of data processing method and device |
| CN107862095B (en) * | 2017-12-07 | 2021-06-01 | 中国银行股份有限公司 | Data processing method and device |
| CN108228730A (en) * | 2017-12-11 | 2018-06-29 | 深圳市买买提信息科技有限公司 | Data lead-in method, device, computer equipment and readable storage medium storing program for executing |
| CN109033184A (en) * | 2018-06-27 | 2018-12-18 | 中国建设银行股份有限公司 | Data processing method and device |
| CN109857832A (en) * | 2019-01-03 | 2019-06-07 | 中国银行股份有限公司 | A kind of preprocess method and device of payment data |
| CN110362617A (en) * | 2019-06-24 | 2019-10-22 | 北京人大金仓信息技术股份有限公司 | Batch data method and system is quickly exported from database based on more concurrent technologies |
| CN112328542A (en) * | 2020-11-25 | 2021-02-05 | 天津凯发电气股份有限公司 | Method for importing data in heterogeneous data file into database |
| WO2022267837A1 (en) * | 2021-06-23 | 2022-12-29 | 比亚迪股份有限公司 | Data archiving method, electronic device, and readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104376082B (en) | 2019-06-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104376082A (en) | Method for importing data in data source file to database | |
| US20220342875A1 (en) | Data preparation context navigation | |
| CN110019218B (en) | Data storage and query method and equipment | |
| CN106445738B (en) | Database backup method and device | |
| US20150205885A1 (en) | CREATING NoSQL DATABASE INDEX FOR SEMI-STRUCTURED DATA | |
| US9256665B2 (en) | Creation of inverted index system, and data processing method and apparatus | |
| CN104615777A (en) | Method and device for real-time data processing based on stream-oriented calculation engine | |
| CN106991100B (en) | Data import method and device | |
| CN106354817B (en) | Log processing method and device | |
| CN103744628A (en) | SSTable file storage method and device | |
| CN106557307B (en) | Service data processing method and system | |
| CN106919697B (en) | Method for simultaneously importing data into multiple Hadoop assemblies | |
| CN109634846B (en) | ETL software testing method and device | |
| CN108121774B (en) | Data table backup method and terminal equipment | |
| CN103778239A (en) | Multi-database data management method and system | |
| US20060149736A1 (en) | Electronic data structure for controlling access to data objects using locks | |
| CN111367890A (en) | Data migration method and device, computer equipment and readable storage medium | |
| CN110889272A (en) | Data processing method, device, equipment and storage medium | |
| CN114297204A (en) | Data storage and retrieval method and device for heterogeneous data source | |
| CN113641651A (en) | Business data management method, system and computer storage medium | |
| CN104933096A (en) | Abnormal key recognition method of database, abnormal key recognition device of database and data system | |
| CN106648550B (en) | Method and device for concurrently executing tasks | |
| CN114090673A (en) | Data processing method, equipment and storage medium for multiple data sources | |
| CN112131291A (en) | JSON data-based structured analysis method, device, equipment and storage medium | |
| CN112579605B (en) | Data storage method, device, storage medium and server |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |