CN101996244A - Device, system and method for inputting batch data into database - Google Patents
Device, system and method for inputting batch data into database Download PDFInfo
- Publication number
- CN101996244A CN101996244A CN2010105360912A CN201010536091A CN101996244A CN 101996244 A CN101996244 A CN 101996244A CN 2010105360912 A CN2010105360912 A CN 2010105360912A CN 201010536091 A CN201010536091 A CN 201010536091A CN 101996244 A CN101996244 A CN 101996244A
- Authority
- CN
- China
- Prior art keywords
- data
- warehouse
- batch
- batch data
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a device, a system and a method for inputting batch data into a database. The device comprises a data analysis unit and a data input unit, wherein the data analysis unit is used for analyzing batch data according to a certain rule and transmitting legal data to the data input unit; and the data input unit is used for inputting the data transmitted by the data analysis unit into the database, recording an input result for inputting the data into the database, and caching unsuccessfully input data so as to be input into the database again when the input is failed. Through the device, the system and the method, the safety and the reliability of batch data input can be increased, and user experience is greatly improved.
Description
Technical field
The present invention relates to the data loading field, particularly a kind of batch data loading device, system and method.
Background technology
The application of database at present more and more widely, the effect of database is also more and more important, lot of data need represent the data of our real needs afterwards by the processing of database, so the more and more necessity that how warehouse-in of batch data safe ready seemed.But present technology is to pursue storage mostly, and to the security of warehouse-in and reliability and the research of in-stockroom operation ease for use seldom.For example, existing warehouse-in mode does not write down and puts in storage once more the data of warehouse-in failure.The warehouse-in efficiency height is no doubt good, in case but loss of data can influence the accuracy that follow-up data is analyzed greatly, and cause analysis result error to occur, consequence is with hardly imaginable.
Summary of the invention
The objective of the invention is to, a kind of batch data loading device is provided, can increase the security and the reliability of batch data warehouse-in, greatly improve user's use experience.
Another object of the present invention is to, provide a kind of batch data to put system in storage, can increase the security and the reliability of batch data warehouse-in, greatly improve user's use experience.
Another purpose of the present invention is, a kind of batch data storage method is provided, and can increase the security and the reliability of batch data warehouse-in, greatly improves user's use experience.
Batch data loading device of the present invention comprises data analysis unit and data loading unit, and wherein, described data analysis unit is used for according to certain rule analysis batch data legal data being sent to the data loading unit; Described data loading unit is used for the data that described data analysis unit is sent are gone into to database, and record data go into the warehouse-in result to database, during for failure, the data of warehouse-in failure is carried out buffer memory in the warehouse-in result, in order to warehouse-in once more.
Wherein, described data analysis unit is further used for illegal data being carried out buffer memory after according to certain rule analysis batch data.
Wherein, described certain rule requires to analyze for the list structure according to tables of data in the database.
Wherein, described data loading unit is further used for going into to database according to the data that predefined storage sends data analysis unit.
Wherein, described data loading unit is further used for putting object information at regular intervals in storage and feeds back to client.
Wherein, described warehouse-in object information comprises batch data storage, batch data warehouse-in physical record number, success/failure record number, warehouse-in failure document storage path.
Batch data of the present invention is put system in storage, comprises the batch data loading device, and further comprise database and client that wherein, described database is used to receive the batch data that the batch data loading device sends; Described client is used to receive the warehouse-in object information that the data loading unit feeds back, to offer the user.
Wherein, described client has visualization interface, and the user is by described visualization interface Real Time Observation warehouse-in object information and the warehouse-in of the batch data in a period of time in the past record.
Batch data storage method of the present invention is batch data being gone into before the database, according to certain rule analysis batch data, with legal data loading, and record data warehouse-in result, when the warehouse-in result is failure, the data of warehouse-in failure are carried out buffer memory, in order to putting in storage once more.
Wherein, comprise the following steps: illegal data to be carried out buffer memory after according to certain rule analysis batch data.
Wherein, described certain rule requires to analyze for the list structure according to tables of data in the database.
The invention has the beneficial effects as follows:, can increase the security and the reliability of batch data warehouse-in according to batch data loading device of the present invention, system and method.In addition, in client, the user can select the data file that will put in storage, can be according to the load setting of system suitable warehouse-in speed simultaneously can video data warehouse-in progress in client, the success ratio of warehouse-in, warehouse-in result and warehouse-in error logging etc., the user can pass through the easy and convenient understanding warehouse situation of client, and all record can be arranged to the warehouse-in in a period of time, any warehouse situation in the user can inquire about during this period of time.In a word, the present invention has very big dirigibility and practicality, has greatly improved the good use experience of user.
Description of drawings
Fig. 1 is the structural representation of batch data loading device of the present invention;
Fig. 2 is the structural representation that batch data of the present invention is put system in storage;
Fig. 3 is the process flow diagram of batch data storage method of the present invention.
Embodiment
Below, 1～3 describe batch data loading device of the present invention, system and method in detail with reference to the accompanying drawings.
As shown in Figure 1, batch data loading device of the present invention, comprise data analysis unit and data loading unit, wherein, data analysis unit is used for analyzing batch data according to certain rule (for example, the list structure according to tables of data in the database requires), legal data are sent to the data loading unit, illegal data are carried out buffer memory.The data loading unit is used for the data that data analysis unit is sent are gone into to database.
In addition, the data loading unit is further used for writing down the data that data analysis unit is sent and goes into warehouse-in result to database, during for failure, the data of warehouse-in failure is carried out buffer memory in the warehouse-in result, in order to warehouse-in once more.
In addition, the data loading unit can also be further used for going into to database according to the data that predefined storage sends data analysis unit; And be further used for to put object information at regular intervals in storage and feed back to client.Wherein, the warehouse-in object information comprises batch data storage, batch data warehouse-in physical record number, success/failure record number, warehouse-in failure document storage path etc.Like this, after batch data was also put in storage by analysis and finished, client can receive the warehouse-in object information of data loading unit feedback, and then, whether the user can will put the data loading of failure in storage or continue other batch data warehouse-ins with decision once more according to the warehouse-in object information in client query.
As shown in Figure 2, batch data of the present invention is put system in storage, is on the basis of above-mentioned batch data loading device, further comprises database and client that wherein, database is used to receive the batch data that the batch data loading device sends; Client is used to receive the warehouse-in object information that the data loading unit feeds back, to offer the user.
When specific implementation, client can be visual client, and the user can be from Real Time Observation warehouse-in object information on the client and the warehouse-in of the batch data in a period of time in the past record.
As shown in Figure 3, batch data storage method of the present invention comprises the following steps:
Step 100: batch data is being gone into before the database, analyzing batch data, legal data are sent to the data loading unit, illegal data are carried out buffer memory according to certain rule (for example, requiring) according to the list structure of tables of data in the database.
Wherein, certain rule requires to analyze for the list structure according to tables of data in the database.
In addition, can further include the following step:
Step 200: record data warehouse-in result, when the warehouse-in result is failure, carry out buffer memory with the data of warehouse-in failure, in order to putting in storage once more.
Method of the present invention can also further be gone into data to database according to predefined storage; Can also further will put object information at regular intervals in storage and feed back to client.Wherein, the warehouse-in object information comprises batch data storage, batch data warehouse-in physical record number, success/failure record number, warehouse-in failure document storage path.
In sum, according to batch data loading device of the present invention, system and method, be on existing batch data warehouse-in basis, the preservation of putting the data integrity of failure in storage is got off, after current warehouse-in finishes, the user can put the data of warehouse-in failure in storage once more again, can increase the security and the reliability of batch data warehouse-in.In addition, in client, the user can select the data file that will put in storage, can be according to the load setting of system suitable warehouse-in speed simultaneously can video data warehouse-in progress in client, the success ratio of warehouse-in, warehouse-in result and warehouse-in error logging etc., the user can pass through the easy and convenient understanding warehouse situation of client, and all record can be arranged to the warehouse-in in a period of time, any warehouse situation in the user can inquire about during this period of time.In a word, the present invention has very big dirigibility and practicality, has greatly improved the good use experience of user.
More than be in order to make those of ordinary skills understand the present invention; and to detailed description that the present invention carried out; but can expect; in the scope that does not break away from claim of the present invention and contained, can also make other variation and modification, these variations and revising all in protection scope of the present invention.
Claims (11)
1. a batch data loading device is characterized in that, comprises data analysis unit and data loading unit, wherein,
Described data analysis unit is used for according to certain rule analysis batch data legal data being sent to the data loading unit;
Described data loading unit is used for the data that described data analysis unit is sent are gone into to database, and record data go into the warehouse-in result to database, during for failure, the data of warehouse-in failure is carried out buffer memory in the warehouse-in result, in order to warehouse-in once more.
2. batch data loading device as claimed in claim 1 is characterized in that, described data analysis unit is further used for illegal data being carried out buffer memory after according to certain rule analysis batch data.
3. batch data loading device as claimed in claim 1 or 2 is characterized in that, described certain rule requires to analyze for the list structure according to tables of data in the database.
4. batch data loading device as claimed in claim 3 is characterized in that, described data loading unit is further used for going into to database according to the data that predefined storage sends data analysis unit.
5. batch data loading device as claimed in claim 4 is characterized in that, described data loading unit is further used for putting object information at regular intervals in storage and feeds back to client.
6. batch data loading device as claimed in claim 5 is characterized in that, described warehouse-in object information comprises batch data storage, batch data warehouse-in physical record number, success/failure record number, warehouse-in failure document storage path.
7. one kind comprises the batch data warehouse-in system of device according to claim 1, it is characterized in that, further comprises database and client, wherein,
Described database is used to receive the batch data that the batch data loading device sends;
Described client is used to receive the warehouse-in object information that the data loading unit feeds back, to offer the user.
8. batch data as claimed in claim 7 is put system in storage, it is characterized in that described client has visualization interface, and the user is by described visualization interface Real Time Observation warehouse-in object information and the warehouse-in of the batch data in a period of time in the past record.
9. batch data storage method, it is characterized in that, batch data is being gone into before the database, according to certain rule analysis batch data, with legal data loading, and record data warehouse-in result, when the warehouse-in result is failure, the data of warehouse-in failure are carried out buffer memory, in order to putting in storage once more.
10. batch data storage method as claimed in claim 9 is characterized in that, comprises the following steps: illegal data to be carried out buffer memory after according to certain rule analysis batch data.
11., it is characterized in that described certain rule requires to analyze for the list structure according to tables of data in the database as claim 9 or 10 described batch data storage methods.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2010105360912A CN101996244A (en) | 2010-11-09 | 2010-11-09 | Device, system and method for inputting batch data into database |
| PCT/CN2010/080273 WO2012062010A1 (en) | 2010-11-09 | 2010-12-24 | Device, system and method for batch data storage |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2010105360912A CN101996244A (en) | 2010-11-09 | 2010-11-09 | Device, system and method for inputting batch data into database |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN101996244A true CN101996244A (en) | 2011-03-30 |
Family
ID=43786398
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2010105360912A Pending CN101996244A (en) | 2010-11-09 | 2010-11-09 | Device, system and method for inputting batch data into database |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN101996244A (en) |
| WO (1) | WO2012062010A1 (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103049533A (en) * | 2012-12-23 | 2013-04-17 | 北京人大金仓信息技术股份有限公司 | Method for quickly loading data into database |
| CN103853719A (en) * | 2012-11-28 | 2014-06-11 | 成都勤智数码科技股份有限公司 | Extensible mass data collection system |
| CN103853713A (en) * | 2012-11-28 | 2014-06-11 | 成都勤智数码科技股份有限公司 | Efficient storage method of mass data |
| CN112835933A (en) * | 2021-02-01 | 2021-05-25 | 河北志晟信息技术股份有限公司 | Socket-based real-time synchronization method for mass data |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101277272A (en) * | 2008-05-16 | 2008-10-01 | 北京航空航天大学 | Method for implementing magnanimity broadcast data warehouse-in |
| CN101556591A (en) * | 2008-04-09 | 2009-10-14 | 北京闻言科技有限公司 | Method for efficiently writing complex contents to database |
| CN101626628A (en) * | 2008-11-28 | 2010-01-13 | 北京杰智讯科技有限公司 | Digital number and web address mapping and pushing system |
| CN101814090A (en) * | 2010-03-22 | 2010-08-25 | 深圳市芯朗金能源技术有限公司 | Batch data insertion method of relational database |
-
2010
- 2010-11-09 CN CN2010105360912A patent/CN101996244A/en active Pending
- 2010-12-24 WO PCT/CN2010/080273 patent/WO2012062010A1/en active Application Filing
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101556591A (en) * | 2008-04-09 | 2009-10-14 | 北京闻言科技有限公司 | Method for efficiently writing complex contents to database |
| CN101277272A (en) * | 2008-05-16 | 2008-10-01 | 北京航空航天大学 | Method for implementing magnanimity broadcast data warehouse-in |
| CN101626628A (en) * | 2008-11-28 | 2010-01-13 | 北京杰智讯科技有限公司 | Digital number and web address mapping and pushing system |
| CN101814090A (en) * | 2010-03-22 | 2010-08-25 | 深圳市芯朗金能源技术有限公司 | Batch data insertion method of relational database |
Non-Patent Citations (2)
| Title |
|---|
| 朱爱华: "广电行业台内分布式异构网络化系统的集成方法探讨和实践", 《电视工程》, 30 June 2008 (2008-06-30), pages 10 - 13 * |
| 李文: "虚拟天文台环境下的海量数据存储与访问技术研究", 《中国优秀硕士学位论文全文数据库》, 15 April 2009 (2009-04-15), pages 138 - 171 * |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103853719A (en) * | 2012-11-28 | 2014-06-11 | 成都勤智数码科技股份有限公司 | Extensible mass data collection system |
| CN103853713A (en) * | 2012-11-28 | 2014-06-11 | 成都勤智数码科技股份有限公司 | Efficient storage method of mass data |
| CN103853713B (en) * | 2012-11-28 | 2018-04-24 | 勤智数码科技股份有限公司 | The efficient storage method of mass data |
| CN103853719B (en) * | 2012-11-28 | 2018-05-22 | 勤智数码科技股份有限公司 | Easily extension mass data collection system |
| CN103049533A (en) * | 2012-12-23 | 2013-04-17 | 北京人大金仓信息技术股份有限公司 | Method for quickly loading data into database |
| CN112835933A (en) * | 2021-02-01 | 2021-05-25 | 河北志晟信息技术股份有限公司 | Socket-based real-time synchronization method for mass data |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2012062010A1 (en) | 2012-05-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2019222261A4 (en) | Cloud based just in time memory analysis for malware detection | |
| US11119843B2 (en) | Verifying application behavior based on distributed tracing | |
| WO2018109589A1 (en) | Updating data records by adding editing functions to non-editable display elements | |
| CN110442712B (en) | Risk determination method, risk determination device, server and text examination system | |
| JP2017033521A (en) | Computer utilization system for integrating and displaying (presenting) heterogeneous information and computer utilization method | |
| US20160080303A1 (en) | Determining topic relevance of an email thread | |
| US9043692B2 (en) | Predictive system recommended actions based on recent activities | |
| RU2016111349A (en) | AUTOMATED RECORDING OF PRODUCTION PROCESS DATA FOR ANALYSIS AND REPORTING ERRORS IN THE DRILLING APPLICATION | |
| US10353760B2 (en) | System verification of interactive screenshots and log files between client systems and server systems within a network computing environment | |
| CN101996244A (en) | Device, system and method for inputting batch data into database | |
| CN105373603A (en) | Method for improving three-layer correlation accuracy | |
| US20180307530A1 (en) | Data persistence method and system thereof in stream computing | |
| US20150312201A1 (en) | Organizing social network messages based on temporal characteristics | |
| CN107770151A (en) | A kind of enterprise's integrated work management system and its method | |
| CN107111649A (en) | User and system data are uploaded into destination locations from source position | |
| CN104461651A (en) | Framework process control method, device and system | |
| CN104598348B (en) | A kind of method and system of the long-range external system interface performance of analysis in real time | |
| KR101694290B1 (en) | Apparatus and method for detecting game bot | |
| US10430454B2 (en) | Systems and methods for culling search results in electronic discovery | |
| Mangler et al. | Magma recharge patterns control eruption styles and magnitudes at Popocatépetl volcano (Mexico) | |
| US8341538B1 (en) | Systems and methods for reducing redundancies in quality-assurance reviews of graphical user interfaces | |
| CN104539449A (en) | Handling method and related device for fault information | |
| US10430140B2 (en) | Method, apparatus and system for opening a web page | |
| US20230113187A1 (en) | Analytics workflow integrated with logic control | |
| WO2022265884A8 (en) | Data governance systems and methods |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C02 | Deemed withdrawal of patent application after publication (patent law 2001) | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20110330 |