CN101556586A - Method, system and device of automatic data collection - Google Patents
Method, system and device of automatic data collection Download PDFInfo
- Publication number
- CN101556586A CN101556586A CNA2008100895425A CN200810089542A CN101556586A CN 101556586 A CN101556586 A CN 101556586A CN A2008100895425 A CNA2008100895425 A CN A2008100895425A CN 200810089542 A CN200810089542 A CN 200810089542A CN 101556586 A CN101556586 A CN 101556586A
- Authority
- CN
- China
- Prior art keywords
- data
- database
- analysis system
- operation analysis
- metadata
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Abstract
The invention discloses a method, a system and a device of automatic data collection. The method has the following steps: a data dispatching task is established; and data is led into a database according to a preset data lead-in condition and the data dispatching task. The embodiment of the invention realizes the purpose of automatically collecting documents issued by a company/group, data of manual collection and the data without a data passage of the source system into the database of the running analysis system to provide the functions of data collection, browsing, summarizing, and the like in an unified interface for the user, and simultaneously can be used for other modules, such as report forms, subject analysis, and the like.
Description
Technical field
The present invention relates to communication technical field, particularly a kind of mthods, systems and devices of automatic data collection.
Background technology
Operation analysis system is an information storage technology, the information processing technology, the achievement that other operation systems of Computer Applied Technology and company/group organically combine, it has effectively utilized BOSS (Business OperationSupport System, business support system) and the bulk information resource that produces of other production systems, the maintenance data storehouse, on-line analytical processing, technology such as data mining, obtain resource, service, the client, finance, professional, basic datas such as network form company information, help company/group grasps market trend, accurately understand rival's situation, identify the market's trend, in time formulate efficient, reasonably marketing strategy guarantees to be in leading position in fierce competitive market.
The file that present company/group issues, manual data such as data of collecting generally all are to offer the final user with the Excel form, and do not have the database of typing operation analysis system to handle automatically, therefore the user uses very inconvenient, require a great deal of time data are carried out artificial treatment, and data integrity and security do not guarantee that it is low especially to carry out efficient yet.
Therefore, the technical scheme of the automatic data collection of data free way in the database of operation analysis system in the file that also company/group is not issued at present, manual data of collecting and the origin system.
Summary of the invention
The embodiment of the invention provides a kind of mthods, systems and devices of automatic data collection, with the database of realizing operation analysis system to data automatic collecting.
For achieving the above object, the embodiment of the invention provides a kind of method of automatic data collection on the one hand, may further comprise the steps:
Set up the data dispatch task;
According to the data importing condition that presets and described data dispatch task with the data importing database.
On the other hand, the embodiment of the invention also provides a kind of operation analysis system, comprising: data copying platform and database,
Described data copying platform is used to set up the data dispatch task, according to the data importing condition that presets and described data dispatch task with the data importing database;
Described database is used to store the data that described data copying platform imports.
On the one hand, the embodiment of the invention also provides a kind of data copying platform, comprising: set up module, be used to set up the data dispatch task again;
First imports module, is used for according to the data importing condition that presets and described data dispatch task the data importing database.
Compared with prior art, the embodiment of the invention has the following advantages: the data copying platform of the embodiment of the invention is set up the data dispatch task, according to the data importing condition that presets and described data dispatch task with the data importing database, thereby the automatic data collection of having realized data free way in the file that company/group is issued, manual data of collecting and the origin system is in the database of operation analysis system.Thereby function such as realize data acquisition for the user provides unified interface, browse, gather, and can use for other modules (as form, subject analysis etc.) simultaneously.
Description of drawings
Fig. 1 is that the overall plan of the embodiment of the invention realizes block diagram;
Fig. 2 is provided with the method flow diagram of data importing condition for the embodiment of the invention;
Fig. 3 is the data dispatch process flow diagram of the embodiment of the invention;
Fig. 4 is the system construction drawing of the operation analysis system of the embodiment of the invention;
Fig. 5 is the structure drawing of device of the data copying platform of the embodiment of the invention.
Embodiment
As shown in Figure 1, be the overall plan realization block diagram of the embodiment of the invention, the embodiment of the invention is set up the data copying platform on the basis of existing operation analysis system.In the embodiment of the invention, the data copying platform data of typing comprise the data of data free way in file, manual data of collecting and the origin system that company/group issues.The data copying platform is according to the data importing condition image data that sets in advance, and with the database of this data importing operation analysis system, the automatic data collection of realizing data free way in the file that company/group is issued, manual data of collecting and the origin system is in the database of operation analysis system.
As shown in Figure 2, for the embodiment of the invention is provided with the method flow diagram of data importing condition, specifically may further comprise the steps:
Step S201, specifying needs to import data source.The data source that needs to import can be the data source of file type, also can be the database of operation system.
Step S202, the essential information in definition of data source.Table name that the specific data source is set up in system and field name, or system generates automatically.
Step S203, the verification rule that setting data is basic.As 11 of cell-phone numbers, time format etc., data are carried out certain cleaning, remove underproof data.
Step S204, the setting data lead-in mode.The data copying platform of the embodiment of the invention can be carried out importing immediately for the smaller data file of data volume, import and can carry out the backstage for the bigger data file of data volume, and the period that can specify a system not to be in a hurry carries out.Cycle imports if desired, then can define the cycle of importing.
As shown in Figure 3, be the data dispatch process flow diagram of the embodiment of the invention.After above-mentioned data importing condition is set up, the data copying platform will start automatic scheduler task, with presetting of task, this task can be regularly or Event triggered will deposit in user FTP (File Transfer Protocol, file transfer protocol (FTP)) file on the server is gathered automatically and content is wherein imported in the operation analysis system data warehouse automatically, or will set up data Automatic Extraction in the database of DBlink in the operation analysis system data warehouse with origin system, make the collection of these data not influenced by human factor, to improve the data acquisition accuracy, promptness specifically may further comprise the steps:
Step S301, a newly-built data scheduler task.After setting the data importing condition, at first data copying platform log-on data scheduling, a newly-built data scheduler task, described data dispatch task can be that periodic scheduling also can be the scheduling of Event triggered.
Step S302, the registration scheduler task.After the newly-built task of data copying platform, should newly-built task registration in system call.
Step S303, data copying platform extracted data.Extracted data in the data of data free way the file that the data copying platform issues from company/group according to registered data dispatch task and the data importing condition that presets, manual data of collecting and the origin system.
Step S304 judges in the metadata of operation analysis system whether have metadata information.The data copying platform needs to call the performance analysis metadata interface before carrying out importing, judge the metadata information of the data of data free way in the file that whether exists company/group to issue in the metadata of operation analysis system, manual data of collecting and the origin system, if there is described metadata information in the metadata of operation analysis system, execution in step S305 then, otherwise execution in step S306.Wherein, metadata is preserved in Query (inquiry) subsystem of operation analysis system, also can be the independent entry data that are independent of outside the Query subsystem.
Step S305 is with the database of data importing operation analysis system.If there is the metadata information of the data of data free way in the file that company/group issues, manual data of collecting and the origin system in the metadata of operation analysis system, then directly with the database of data importing operation analysis system.
Step S306 imports metadata with metadata information.If there is not the metadata information of the data of data free way in the file that company/group issues, manual data of collecting and the origin system in the metadata of operation analysis system, then described metadata information is imported metadata, and execution in step S305.
Judge in the foregoing description that the step that whether has metadata information in the metadata of operation analysis system also can be placed on and carry out after the data importing condition is set.
The whole gatherer process of the foregoing description automatically performs fully, and without manual intervention, the efficient of execution is higher.Data through the typing of data copying platform, because having generated can be by the discernible metadata information of operation analysis system, so can use the analytic function in the operation analysis system, data are further processed, be very easy to the business personnel and handle the outer data of operation analysis system.For example the business personnel of market department A logins operation analysis system, open the data copying platform, the portion's catalogue that comes into the market is created " refuse messages information " catalogue, under " refuse messages information " catalogue, create a refuse collection module, the needed table information of definition of data typing.Then, the task of collecting is handed down to the business hall, the business hall personnel download " refuse messages information " form that needs typing, insert information, then by the data copying platform, are entered in the operation analysis system oneself collecting good information.The business personnel A of market department by the analytic function in the operation analysis system, gathers the data of " refuse messages information ", analyzes, and forms analysis report, reports to the Marketing Manager.
If business personnel A finds every month and will do these work, so he becomes the task in cycle to these active configuration, the system registry task is gathered these data automatically by the data copying platform.
As shown in Figure 4, the structural drawing for the operation analysis system of the embodiment of the invention comprises: data copying platform 1, database 2.
Above-mentioned data copying platform 1 is used to set up the data dispatch task, according to the data importing condition that presets and described data dispatch task with data importing database 2.
Above-mentioned database 2 is used to store the data that data copying platform 1 imports.
As shown in Figure 5, the structure drawing of device for the data copying platform of the embodiment of the invention comprises: set up module 11, be used to set up the data dispatch task.First imports module 12, is used for according to the data importing condition that presets and described data dispatch task data importing database 2.
Above-mentioned data typing puts down 1, also comprises: module 13 is set, is used to be provided with the data importing condition.Judge module 14, whether the metadata that is used for judging operation analysis system exists the metadata information of described data.Generation module 15 is used for generating the metadata information of described data when there is not the metadata information of described data in the metadata of above-mentioned operation analysis system.Second imports module 16, when being used for the metadata information when the described data of generation module 15 generations, described metadata information is imported in the described metadata, and with the database 2 of described data importing operation analysis system.
The above-mentioned module 13 that is provided with further comprises: specify submodule, be used to specify the data source that needs to import data.Define submodule, be used to define the essential information of described data source.Set submodule, be used to set verification rule and the lead-in mode that described needs import data.
It will be appreciated by those skilled in the art that the module in the device among the embodiment can be distributed in the device of embodiment according to the embodiment description, also can carry out respective change and be arranged in the one or more devices that are different from present embodiment.The module of the foregoing description can be merged into a module, also can further split into a plurality of submodules.
The data copying platform of the embodiment of the invention is set up the data dispatch task, according to the data importing condition that presets and described data dispatch task with the data importing database, thereby the automatic data collection of having realized data free way in the file that company/group is issued, manual data of collecting and the origin system is in the database of operation analysis system.Thereby function such as realize data acquisition for the user provides unified interface, browse, gather, and can use for other modules (as form, subject analysis etc.) simultaneously.
Through the above description of the embodiments, those skilled in the art can be well understood to the present invention and can realize by hardware, certainly also can realize, but the former is better embodiment under a lot of situation by the mode that software adds essential general hardware platform.Based on such understanding, the part that technical scheme of the present invention contributes to prior art in essence in other words can embody with the form of hardware product.
More than disclosed only be several specific embodiment of the present invention, still, the present invention is not limited thereto, any those skilled in the art can think variation all should fall into protection scope of the present invention.
Claims (10)
1, a kind of method of automatic data collection is characterized in that, may further comprise the steps:
Set up the data dispatch task;
According to the data importing condition that presets and described data dispatch task with the data importing database.
2, the method for automatic data collection according to claim 1 is characterized in that, also comprises described data importing condition is set, and comprising:
Specify the data source that needs to import data;
Define the essential information of described data source;
Set described verification rule and the lead-in mode that needs to import data.
3, the method for automatic data collection according to claim 1 is characterized in that described data dispatch task comprises periodic data dispatch task.
4, the method for automatic data collection according to claim 1 is characterized in that described database is the database of operation analysis system, before described database with the described operation analysis system of data importing, also comprises:
Judge the metadata information that whether has described data in the metadata of operation analysis system;
The metadata information that has described data in the metadata of described operation analysis system is then with the database of described data importing operation analysis system;
Otherwise, generate the metadata information of described data, described metadata information is imported in the described metadata, and with the database of described data importing operation analysis system.
5,, it is characterized in that the data of data free way in the described data file that issues that comprises company/group, manual data of collecting and the origin system as the method for automatic data collection as described in claim 1 or 3.
6, a kind of operation analysis system is characterized in that, comprising: data copying platform and database,
Described data copying platform is used to set up the data dispatch task, according to the data importing condition that presets and described data dispatch task with the data importing database;
Described database is used to store the data that described data copying platform imports.
7, a kind of data copying platform is characterized in that, comprising:
Set up module, be used to set up the data dispatch task;
First imports module, is used for according to the data importing condition that presets and described data dispatch task the data importing database.
8, as data copying platform as described in the claim 7, it is characterized in that, also comprise:
Module is set, is used to be provided with the data importing condition.
9, as data copying platform as described in the claim 8, it is characterized in that the described module that is provided with comprises:
Specify submodule, be used to specify the data source that needs to import data;
Define submodule, be used to define the essential information of described data source;
Set submodule, be used to set verification rule and the lead-in mode that described needs import data.
10, as data copying platform as described in the claim 7, it is characterized in that described database is the database of operation analysis system, described data copying platform also comprises:
Judge module, whether the metadata that is used for judging operation analysis system exists the metadata information of described data;
Generation module is used for generating the metadata information of described data when there is not the metadata information of described data in the metadata of described operation analysis system;
Second imports module, be used for when described generation module generates the metadata information of described data, described metadata information being imported in the described metadata, and with the database of described data importing operation analysis system.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CNA2008100895425A CN101556586A (en) | 2008-04-07 | 2008-04-07 | Method, system and device of automatic data collection |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CNA2008100895425A CN101556586A (en) | 2008-04-07 | 2008-04-07 | Method, system and device of automatic data collection |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN101556586A true CN101556586A (en) | 2009-10-14 |
Family
ID=41174704
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CNA2008100895425A Pending CN101556586A (en) | 2008-04-07 | 2008-04-07 | Method, system and device of automatic data collection |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN101556586A (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102222120A (en) * | 2011-06-16 | 2011-10-19 | 河南新创元信息网络有限公司 | Method and system for quickly collecting binary information of dispersion demand |
| CN102722587A (en) * | 2012-06-12 | 2012-10-10 | 苏州微逸浪科技有限公司 | Method for processing passive receiving on basis of custom task scheduling |
| CN103049550A (en) * | 2012-12-27 | 2013-04-17 | 北京思特奇信息技术股份有限公司 | Method and system for importing common files to database |
| CN104268172A (en) * | 2014-09-15 | 2015-01-07 | 北京京东尚科信息技术有限公司 | Data extraction method and device |
| CN104508660A (en) * | 2012-06-04 | 2015-04-08 | 惠普发展公司，有限责任合伙企业 | User-defined loading of data onto a database |
| CN106227806A (en) * | 2016-07-22 | 2016-12-14 | 浪潮电子信息产业股份有限公司 | A kind of service report system based on corporate client |
| CN106649416A (en) * | 2015-11-07 | 2017-05-10 | 上海海典软件股份有限公司 | Data analysis method for drugstore dynamic reports |
| CN107368593A (en) * | 2017-07-25 | 2017-11-21 | 万帮充电设备有限公司 | Data lead-in method, device and server |
| CN112988730A (en) * | 2021-03-29 | 2021-06-18 | 国网宁夏电力有限公司电力科学研究院 | Metadata collection method based on enterprise data inventory |
| CN113094406A (en) * | 2019-12-23 | 2021-07-09 | 内蒙古电力（集团）有限责任公司电力营销服务与运营管理分公司 | Power marketing data management method and system |
-
2008
- 2008-04-07 CN CNA2008100895425A patent/CN101556586A/en active Pending
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102222120A (en) * | 2011-06-16 | 2011-10-19 | 河南新创元信息网络有限公司 | Method and system for quickly collecting binary information of dispersion demand |
| US10474658B2 (en) | 2012-06-04 | 2019-11-12 | Micro Focus Llc | User-defined loading of data onto a database |
| CN104508660A (en) * | 2012-06-04 | 2015-04-08 | 惠普发展公司，有限责任合伙企业 | User-defined loading of data onto a database |
| CN102722587A (en) * | 2012-06-12 | 2012-10-10 | 苏州微逸浪科技有限公司 | Method for processing passive receiving on basis of custom task scheduling |
| CN103049550A (en) * | 2012-12-27 | 2013-04-17 | 北京思特奇信息技术股份有限公司 | Method and system for importing common files to database |
| CN104268172A (en) * | 2014-09-15 | 2015-01-07 | 北京京东尚科信息技术有限公司 | Data extraction method and device |
| CN104268172B (en) * | 2014-09-15 | 2018-06-26 | 北京京东尚科信息技术有限公司 | The method and apparatus for extracting data |
| CN106649416A (en) * | 2015-11-07 | 2017-05-10 | 上海海典软件股份有限公司 | Data analysis method for drugstore dynamic reports |
| CN106227806A (en) * | 2016-07-22 | 2016-12-14 | 浪潮电子信息产业股份有限公司 | A kind of service report system based on corporate client |
| CN107368593A (en) * | 2017-07-25 | 2017-11-21 | 万帮充电设备有限公司 | Data lead-in method, device and server |
| CN107368593B (en) * | 2017-07-25 | 2020-09-01 | 万帮充电设备有限公司 | Data import method and device and server |
| CN113094406A (en) * | 2019-12-23 | 2021-07-09 | 内蒙古电力（集团）有限责任公司电力营销服务与运营管理分公司 | Power marketing data management method and system |
| CN112988730A (en) * | 2021-03-29 | 2021-06-18 | 国网宁夏电力有限公司电力科学研究院 | Metadata collection method based on enterprise data inventory |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101556586A (en) | Method, system and device of automatic data collection | |
| CN110245035A (en) | A kind of link trace method and device | |
| CN106778253A (en) | Threat context aware information security Initiative Defense model based on big data | |
| CN106295382B (en) | A kind of Information Risk preventing control method and device | |
| US20160171505A1 (en) | Extract, transform, and load (etl) processing | |
| CN107103064B (en) | Data statistical method and device | |
| CN111666490A (en) | Information pushing method, device, equipment and storage medium based on kafka | |
| CN112632135A (en) | Big data platform | |
| CN102833111B (en) | A kind of visual HTTP data monitoring and managing method and device | |
| CN102508919A (en) | Data processing method and system | |
| CN109408541A (en) | Report decomposes statistical method, system, computer equipment and storage medium | |
| CN111461650B (en) | Schedule reminding method and device, storage medium and intelligent equipment | |
| CN110928681A (en) | Data processing method and device, storage medium and electronic device | |
| CN202145321U (en) | To-be processed information cue and mobile office integrated system | |
| CN111048164A (en) | Medical big data long-term storage system | |
| CN103678425A (en) | Integrated analysis for multiple systems | |
| CN112181678A (en) | Service data processing method, device and system, storage medium and electronic device | |
| CN111949772A (en) | Intelligent customer service and knowledge base system and management method | |
| CN109558403A (en) | Data aggregation method and device, computer installation and computer readable storage medium | |
| CN201307870Y (en) | Phone bill distributed type searching engine system | |
| CN116263717A (en) | Order service processing method and device based on event | |
| CN102708502A (en) | XPDL (extensible markup language process definition language)-based multi-step marketing process framework agreement | |
| CN111368179A (en) | Method for realizing general operation prompt configuration and display control in cloud platform process application business approval link | |
| CN115168297A (en) | Bypassing log auditing method and device | |
| CN111475505A (en) | Data acquisition method and equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C12 | Rejection of a patent application after its publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20091014 |