CN201654777U - Dynamic data acquisition device - Google Patents
Dynamic data acquisition device Download PDFInfo
- Publication number
- CN201654777U CN201654777U CN200920151677XU CN200920151677U CN201654777U CN 201654777 U CN201654777 U CN 201654777U CN 200920151677X U CN200920151677X U CN 200920151677XU CN 200920151677 U CN200920151677 U CN 200920151677U CN 201654777 U CN201654777 U CN 201654777U
- Authority
- CN
- China
- Prior art keywords
- unit
- management
- dynamic data
- acquisition
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Lifetime
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The utility model discloses a dynamic data acquisition device, belonging to the technical field of distributed dynamic data acquisition. The dynamic data acquisition device comprises a dispatching management unit, a dispatching control unit, an acquisition unit and a releasing unit, wherein the dispatching management unit is respectively connected with the dispatching control unit, the acquisition unit and the releasing unit, and used for receiving a thesaurus to be acquired or an acquired retrieval conditions, creating acquisition tasks and dynamic acquisition strategies, and storing acquisition results; the dispatching control unit is respectively connected with the dispatching management unit and the acquisition unit, and used for analyzing the acquisition tasks created by the dispatching management unit, and controlling the acquisition unit to conduct acquisition; the acquisition unit is respectively connected with the dispatching control unit and the dispatching management unit, and used for acquiring dynamic data, and sending the acquired data to the dispatching management unit for processing and storage; and the releasing unit is connected with the scheduling management unit and used for releasing the acquisition results filtered and stored by the dispatching management unit.
Description
Technical field
The utility model relates to a kind of towards the internet " darknet " (Deep Web) dynamic data distribution of resource formula harvester, belong to distributed dynamic acquisition technique field.
Background technology
For the Competitive Intelligence field, the user need be from seeking own useful or disadvantageous information in the internet arena widely, and be difficult to reach user's purpose by common search engine, one of reason is that search engine disperses to cause being difficult to obtain complete data, the 2nd, search engine can only obtain static web data, and can not obtain dynamic data, can not obtain data by query interfaces such as search engines, more can not obtain the business data of inside data of enterprise or purchase, these data all are Deep Web data.And static web data has just accounted for the very little part of whole Web data, can not satisfy user's demand far away.
For the academic research field, the scope that the user wishes to obtain resource is extensive more good more, professional more good more.And the data of professional domain are to belong to Deep Web data mostly, often can not obtain by common search engine, but by obtaining from database construction, the search engine of specialty or the business database of purchase.
The utility model content
For solving the problem that prior art exists, the purpose of this utility model is to provide a kind of Dynamic Data Acquiring device, can carry out dynamic strategy to acquisition tasks and formulate, balance dispatching, logic control are gathered distributed terminator, thereby obtain internet deep layer dynamic data.
The technical scheme of Dynamic Data Acquiring device described in the utility model is as follows:
Described Dynamic Data Acquiring device, comprise management and running unit, scheduling controlling unit, collecting unit and release unit, wherein said management and running unit is connected with scheduling controlling unit, collecting unit and release unit respectively, be used to receive the dictionary of needs collection or the search condition of collection, create acquisition tasks and dynamic acquisition strategy, and the storage of collected result; Described scheduling controlling unit is connected with collecting unit with the management and running unit respectively, is used for the acquisition tasks that the scheduling administrative unit is created is analyzed, and controls described collecting unit and gather; Described collecting unit is connected with the management and running unit with the scheduling controlling unit respectively, is used to gather dynamic data; And described release unit is connected with the management and running unit, is used to issue the collection result after the management and running unit filters and stores.
Wherein, described management and running unit also comprises management and running subelement of gathering dictionary management subelement that required dictionary manages, acquisition tasks being configured and managing and the storing sub-units that is used for the storage of collected result.
Further, described dictionary management subelement comprises that dictionary sets up module, dictionary complementary module and search condition module.
Described management and running subelement comprises the creation task module, selects Data Source module, policy development and change module and filter to store module is set.
And described storing sub-units comprises index module and topic module.
Described scheduling controlling unit comprises analyzes subelement and distributed scheduling control sub unit.
Preferably, described scheduling controlling unit is a distributed load equalizing scheduling logic controller;
Described collecting unit comprises towards the automation collection device of WEB resource and towards the automation collection device of protocol resource;
Described release unit is to gather storehouse release management platform.
Further, described management and running subelement is the acquisition tasks dispatching management information system;
Described dictionary management subelement is to gather the mimetic word base management system;
Described storing sub-units is to gather the memory scheduling system.
And described index submodule is the index data library module;
Described theme submodule is the subject data library module.
Specifically, described dynamic data comprises dynamic web page data, business data, open storage data and inside data of enterprise.
Dynamic Data Acquiring device described in the utility model can carry out dynamic strategy to acquisition tasks and formulate, and balance dispatching, logic control are gathered distributed terminator, thereby obtain internet deep layer dynamic data.
Description of drawings
Fig. 1 is the structural representation of the utility model Dynamic Data Acquiring device;
Fig. 2 is the another kind of structural representation of the utility model Dynamic Data Acquiring device;
Fig. 3 is the structural representation of the utility model management and running unit;
Fig. 4 is the most preferred embodiment structural representation of the utility model Dynamic Data Acquiring device;
Fig. 5 is a surface chart of creating acquisition tasks in the utility model most preferred embodiment;
Fig. 6 is a surface chart of selecting data resource in the utility model most preferred embodiment;
Fig. 7 is a surface chart of formulating acquisition strategies in the utility model most preferred embodiment;
Fig. 8 is the program run figure that gathers resources bank in the utility model most preferred embodiment;
Fig. 9 is the surface chart of issue image data in the utility model most preferred embodiment.
Embodiment
The utility model provides a kind of Dynamic Data Acquiring device, described device is different towards static Web page link analysis mechanism with universal search engine, function with balance dispatching, logic control collection internet deep layer dynamic data, can be according to user's search condition, create different acquisition tasks, formulate different dynamic acquisition strategies, realize quick collection dynamic data.
Below by specific embodiment described Dynamic Data Acquiring device is described.
As shown in Figure 1, Dynamic Data Acquiring device described in the utility model comprises management and running unit, scheduling controlling unit, collecting unit and release unit, described management and running unit is used to import theme dictionary or search condition, establishment acquisition tasks, acquisition strategies, analyze by described scheduling controlling unit, and control described collecting unit and gather, collection result is issued by described release unit after filtering storage by the management and running unit.
In the device described in the utility model, management and running unit and scheduling controlling unit are core cells.Wherein the management and running unit is used to import theme dictionary or search condition, formulation Dynamic Data Acquiring task, and the dynamic data that collects is filtered and stores.In gatherer process, can end, suspend and revise acquisition strategies according to user's needs.The scheduling controlling unit is used for acquisition tasks and strategy are carried out the intelligent algorithm analysis, and the logic control collecting unit is gathered distributed terminator.
Secondly, collecting unit is used for the dynamic data of distributed server system is gathered; Release unit is used for collection result is issued.
As shown in Figure 2, in described Dynamic Data Acquiring device, the management and running unit further comprises dictionary management subelement, management and running subelement and storing sub-units.Wherein, as shown in Figure 3, dictionary management subelement comprises that dictionary sets up module, dictionary complementary module and search condition module, and wherein dictionary is set up module and is used to import the theme dictionary; Described dictionary complementary module is used for the theme dictionary is replenished; Described search condition module is used to import search condition.The management and running subelement also comprises the creation task module, selects Data Source module, policy development and change module and filter to store module is set.Described management and running subelement has customization function, can create acquisition tasks according to user's needs, select Data Source, filtercondition is set, storage mode and formulate acquisition strategies etc., and can carry out the change of strategy according to the feedback information of scheduling controlling unit; Storing sub-units comprises index module and topic module, is used for the data message of collecting unit collection is stored.
As shown in Figure 2, in described Dynamic Data Acquiring device, the scheduling controlling unit further comprises analyzes subelement and distributed scheduling control sub unit.Wherein, described analysis subelement is used for various tasks that the scheduling administrative unit is formulated and strategy etc. and carries out the intelligent algorithm analysis; The distributed scheduling control sub unit is used for according to the analysis result of analyzing subelement, and balance dispatching logic control collecting unit carries out data acquisition to distributed terminator.
Summarize in fact, the principle of Dynamic Data Acquiring device described in the utility model is as follows:
Theme dictionary or search condition according to user's input generate corresponding search condition, and according to acquisition tasks, Data Source and acquisition strategies, distributed terminator are carried out analog access and resolves the data message that returns; The filtering rule of setting according to the user filters the data message that returns; Information after the filtration is stored according to form and setting that the user sets, and carries out field index according to the index setting and handle; In the gatherer process, the user can adjust acquisition strategies, for example can adjust time, cycle, the storage policy of collection, can also suspend or restart the subtask of collection etc.; The result of Cai Jiing (comprising schedule information) issues by release unit at last.
In sum, described Dynamic Data Acquiring device has been realized the collection to internet deep layer dynamic data by distributed terminator is carried out balance dispatching and logic control.
Below in conjunction with reality, describe most preferred embodiment of the present utility model in detail.
In the utility model most preferred embodiment, distributed terminator adopts the distributed server Database Systems, and in the Dynamic Data Acquiring device is installed according to user's request pre-configured all Data Sources that can gather tabulations.
As shown in Figure 4, the utility model most preferred embodiment adopts following set-up mode:
The scheduling controlling unit is a distributed load equalizing scheduling logic controller, and distributed load equalizing scheduling logic controller is used for based on the intellectual analysis algorithm acquisition tasks and scheduling strategy being analyzed, and balance dispatching, logic control are carried out in collection; Described distributed load equalizing scheduling logic controller can be arranged in different main frames for a plurality of, also can carry out the rank setting, the function that expansion is gathered.
Collecting unit comprises towards the automation collection device of Web resource and towards the automation collection device of protocol resource, described towards the Web resource the automation collection device and be respectively a plurality of towards the automation collection device of protocol resource, be used for distributed terminator is carried out data acquisition.
Release unit is to gather storehouse release management platform, and main what adopt is that storehouse release management platform gathered in self-built theme, is used for the data message of gathering is carried out Classification Management and uses issue.
Secondly, the management and running subelement is the acquisition tasks dispatching management information system, adopts human-computer interaction interface, and acquisition tasks, storage setting, acquisition strategies etc. can be set; Dictionary management subelement is to gather the mimetic word base management system, is used to import theme dictionary or search condition, according to filtercondition, the feature dictionary of the data message gathered is filtered, and original dictionary is replenished; Storing sub-units is to gather the memory scheduling system, comprises index data base and subject data base.
The use step of the utility model most preferred embodiment is as follows:
Specifically, the dictionary of input needs collection is the data acquisition function according to theme.The user can pass through the theme dictionary of the mode specified data collection of interface typing or file importing, and acquisition system is gathered according to dictionary automatically.
The user according to the navigation feature of system, creates acquisition tasks, as shown in Figure 5 according to the demand of oneself; Select Data Source, form, as shown in Figure 6; Select to carry out time, cycle and the acquisition strategies of acquisition tasks, as shown in Figure 7; Select data storage destination address, storage format, index setting and filtercondition etc.
Step 3, execution acquisition tasks start acquisition server and gather.As shown in Figure 8.
Distributed load equalizing scheduling logic controller carries out intellectual analysis to above-mentioned acquisition tasks according to the strategy of setting, and starts the collection that the automation collection device carries out dynamic data automatically.Filtering rule and address stored and formats stored that the data based acquisition tasks dispatching system of gathering is set send to collection memory scheduling system and store.Gather the mimetic word base management system and gather out the feature dictionary of dynamic data, the feature speech that does not wherein have is replenished.
In the gatherer process; the user can check the progress of acquisition tasks at any time by the acquisition tasks dispatching system; suspend or restart the subtask of collection, time that adjustment is gathered and acquisition strategies etc., distributed balance dispatching logic controller is dispatched based on the intellectual analysis algorithm.
Wherein, in step 1, in gathering simulation descriptor base management system, also can import specific search condition, the specific search condition that input is gathered is the data acquisition function according to search condition.Described search condition can be a logical expression, it also can be a simple natural descriptive statement, system can resolve input rule automatically, such as need from the full text field of data source, obtaining the information that comprises " certain name of product ", and do not need to comprise that the information of " certain Business Name ", acquisition system resolves and generate corresponding search condition automatically as full text=" certain name of product " and not full text=" certain Business Name ".
In sum, Dynamic Data Acquiring device described in the utility model, customization function is abundant, realizes the collection to the fast and flexible of dynamic data.For example; the user is by the acquisition tasks dispatching management information system; can gather data selected theme, search condition etc.; can select Data Source; also can select field mappings relation to come source field to collect simultaneously in the field or the like, can formulate strategy, time and the cycle etc. of collection a plurality of, and in gatherer process; can change, suspend or end acquisition tasks at any time, can carry out classification and storage according to storage format, address and the index field set.
And, Dynamic Data Acquiring device described in the utility model, expandability and robustness is good.The distributed capture function of existing individual task has the expansion of system level again, can be deployed to a plurality of main frames to distributed load equalizing scheduling logic controller, has expanded the distributed capture function.
In addition, characteristics of the present utility model also comprise practicality easy and simple to handle.The man-machine interaction performance is good, and the user only need be according to the prompting of system, and step-by-step the setting gets final product, such as in management system, importing acquisition tasks according to system's navigation hint, select the acquisition tasks data source, optional simple filtering condition options after accomplishing the setting up starts collector and gets final product.
Hardware components of the present utility model comprises:
1) at least one mainstream configuration commercial server
The hardware basic configuration is 4CPU, 8G internal memory, 1T disk space, comprise prepackage Windows2003/2000Server operating system and ASP.NET application server, 3 of virtual opetrating systems also can be expanded, and are used for the deployment of other distributed capture system of operating system grade; Adopt other distributed capture function of server level, then can expand to many Tianwan businessmans as required and use server.
2) at least 3 data storage servers
With data storage and the database storage system that is integrated into the center, mass memory and full-text index function relationship type Database Systems are supported in prepackage, such as the Kbase system of CNKI company.
3) at least one data directory server
The index information of main storage of collected data, purpose are to accelerate data integration and retrieval rate and efficient.
4) at least 1 file server
The information of the file layout of the main storage of collected of file server, file layout comprise information such as picture, webpage.
The utility model has comprised multiple technologies, comprise acquisition technique towards the WEB resource, at the page analysis technology of WEB resource and extraction technique as a result, towards the acquisition technique of various protocols mode (comprising XML, HTTP or the like) access resources, based on the distributed scheduling technology of intellectual analysis and data directory memory technology etc.
Though above at the principle and the embodiment of Dynamic Data Acquiring device; but; under above-mentioned guiding of the present utility model; those skilled in the art can carry out various improvement and distortion on the basis of the foregoing description, and these improvement or distortion drop in the protection domain of the present utility model.It will be understood by those skilled in the art that top specific descriptions just in order to explain the purpose of this utility model, are not to be used to limit the utility model.Therefore, thought of the present utility model is not limited to the embodiment of above explanation, and thought category of the present utility model not only comprises the scope of claims record, also comprises with claim being equal to or distortion of equal value.
Claims (10)
1. a Dynamic Data Acquiring device is characterized in that: comprise management and running unit, scheduling controlling unit, collecting unit and release unit, wherein
Described management and running unit is connected with scheduling controlling unit, collecting unit and release unit respectively, is used to receive the dictionary of needs collection or the search condition of collection, creates acquisition tasks and dynamic acquisition strategy, and the storage of collected result;
Described scheduling controlling unit is connected with collecting unit with the management and running unit respectively, is used for the acquisition tasks that the scheduling administrative unit is created is analyzed, and controls described collecting unit and gather;
Described collecting unit is connected with the management and running unit with the scheduling controlling unit respectively, is used to gather dynamic data, and the data that collection is obtained are sent to the management and running unit handles and store; And
Described release unit is connected with the management and running unit, is used to issue the collection result after the management and running unit filters and stores.
2. Dynamic Data Acquiring device according to claim 1 is characterized in that: described management and running unit also comprises management and running subelement of gathering dictionary management subelement that required dictionary manages, acquisition tasks being configured and managing and the storing sub-units that is used for the storage of collected result.
3. Dynamic Data Acquiring device according to claim 2 is characterized in that:
Described dictionary management subelement comprises that also dictionary sets up module, dictionary complementary module and search condition module.
4. Dynamic Data Acquiring device according to claim 2 is characterized in that:
Described management and running subelement comprises the creation task module, selects Data Source module, policy development and change module and filter to store module is set.
5. Dynamic Data Acquiring device according to claim 2 is characterized in that:
Described storing sub-units comprises index module and topic module.
6. Dynamic Data Acquiring device according to claim 1 is characterized in that:
Described scheduling controlling unit comprises analyzes subelement and distributed scheduling control sub unit.
7. Dynamic Data Acquiring device according to claim 6 is characterized in that:
Described scheduling controlling unit is a distributed load equalizing scheduling logic controller;
Described collecting unit comprises towards the automation collection device of Web resource and towards the automation collection device of protocol resource;
Described release unit is to gather storehouse release management platform.
8. Dynamic Data Acquiring device according to claim 7 is characterized in that:
Described management and running subelement is the acquisition tasks dispatching management information system;
Described dictionary management subelement is to gather the mimetic word base management system;
Described storing sub-units is to gather the memory scheduling system.
9. Dynamic Data Acquiring device according to claim 8 is characterized in that:
Described index submodule is the index data library module;
Described theme submodule is the subject data library module.
10. Dynamic Data Acquiring device according to claim 9 is characterized in that:
Described dynamic data comprises dynamic web page data, business data, open storage data and inside data of enterprise.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN200920151677XU CN201654777U (en) | 2009-04-22 | 2009-04-22 | Dynamic data acquisition device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN200920151677XU CN201654777U (en) | 2009-04-22 | 2009-04-22 | Dynamic data acquisition device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN201654777U true CN201654777U (en) | 2010-11-24 |
Family
ID=43120020
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN200920151677XU Expired - Lifetime CN201654777U (en) | 2009-04-22 | 2009-04-22 | Dynamic data acquisition device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN201654777U (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101840432A (en) * | 2010-05-11 | 2010-09-22 | 同方知网(北京)技术有限公司 | Data mining device based on Deep Web deep dynamic data and method thereof |
| CN102426575A (en) * | 2011-07-28 | 2012-04-25 | 中兴通讯股份有限公司 | IPTV (Internet Protocol Television) data acquisition method and system |
| CN104537288A (en) * | 2015-01-30 | 2015-04-22 | 武汉虹旭信息技术有限责任公司 | Intelligent feedback type generated high-frequency dictionary distributed system and method thereof |
| CN104980517A (en) * | 2015-06-26 | 2015-10-14 | 深圳市腾讯计算机系统有限公司 | Snapshot-based cluster sensing system, method and device |
| CN106603292A (en) * | 2016-12-19 | 2017-04-26 | 国云科技股份有限公司 | Network resource data acquisition system and realization method thereof |
| CN110362020A (en) * | 2019-08-28 | 2019-10-22 | 四川省东宇信息技术有限责任公司 | Intelligent three-dimensional posture and vibration monitor and its monitoring method |
| CN114692057A (en) * | 2022-05-31 | 2022-07-01 | 深圳市房帮帮互联网科技有限公司 | Online service platform for potential user tracking based on user browsing volume |
-
2009
- 2009-04-22 CN CN200920151677XU patent/CN201654777U/en not_active Expired - Lifetime
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101840432A (en) * | 2010-05-11 | 2010-09-22 | 同方知网(北京)技术有限公司 | Data mining device based on Deep Web deep dynamic data and method thereof |
| CN101840432B (en) * | 2010-05-11 | 2012-07-25 | 同方知网(北京)技术有限公司 | Data mining device based on Deep Web deep dynamic data and method thereof |
| CN102426575A (en) * | 2011-07-28 | 2012-04-25 | 中兴通讯股份有限公司 | IPTV (Internet Protocol Television) data acquisition method and system |
| CN104537288A (en) * | 2015-01-30 | 2015-04-22 | 武汉虹旭信息技术有限责任公司 | Intelligent feedback type generated high-frequency dictionary distributed system and method thereof |
| CN104537288B (en) * | 2015-01-30 | 2017-07-28 | 武汉虹旭信息技术有限责任公司 | Intelligent feedback generates the distributed system and its method of high frequency dictionary |
| CN104980517A (en) * | 2015-06-26 | 2015-10-14 | 深圳市腾讯计算机系统有限公司 | Snapshot-based cluster sensing system, method and device |
| CN104980517B (en) * | 2015-06-26 | 2018-04-06 | 深圳市腾讯计算机系统有限公司 | Cluster sensory perceptual system based on snapshot, method and apparatus |
| CN106603292A (en) * | 2016-12-19 | 2017-04-26 | 国云科技股份有限公司 | Network resource data acquisition system and realization method thereof |
| CN106603292B (en) * | 2016-12-19 | 2019-12-10 | 国云科技股份有限公司 | network resource data acquisition system and implementation method thereof |
| CN110362020A (en) * | 2019-08-28 | 2019-10-22 | 四川省东宇信息技术有限责任公司 | Intelligent three-dimensional posture and vibration monitor and its monitoring method |
| CN114692057A (en) * | 2022-05-31 | 2022-07-01 | 深圳市房帮帮互联网科技有限公司 | Online service platform for potential user tracking based on user browsing volume |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101840432B (en) | Data mining device based on Deep Web deep dynamic data and method thereof | |
| CN201654777U (en) | Dynamic data acquisition device | |
| CN100452036C (en) | Contents management system, contents management method, and computer program | |
| CN102822819B (en) | Management data inquiry | |
| WO2008007382A3 (en) | System and method for event management | |
| CN102446225A (en) | Real-time search method, device and system | |
| US20150356137A1 (en) | Systems and Methods for Optimizing Data Analysis | |
| CN110417873B (en) | Network information extraction system for realizing recording webpage interactive operation | |
| CN108509437A (en) | A kind of ElasticSearch inquiries accelerated method | |
| CN102117331B (en) | Video search method and system | |
| CN102710795A (en) | Hotspot collecting method and device | |
| WO2014099488A1 (en) | Personalized search library based on continual concept correlation | |
| CN102521232A (en) | Distributed acquisition and processing system and method of internet metadata | |
| CN109634652A (en) | A kind of method, apparatus of data processing, computer storage medium and terminal | |
| CN105007314A (en) | Big data processing system oriented to mass reading data of readers | |
| CN111159133A (en) | Distributed forum system based on micro-service | |
| JP6262505B2 (en) | Distributed data virtualization system, query processing method, and query processing program | |
| CN102024037B (en) | A kind of search method of massive image data of website operation system | |
| CN106484832A (en) | Assume the method and system of project set | |
| WO2016206395A1 (en) | Weekly report information processing method and device | |
| CN101923579B (en) | File management method based on context flow | |
| CN106649462A (en) | Implementation method for mass data full-text retrieval scene | |
| CN102184173A (en) | Method for searching Internet data | |
| CN113553425A (en) | Data aggregation method, device, equipment and storage medium based on RPA and AI | |
| JP5950522B2 (en) | Apparatus, method, and program for displaying document list |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CX01 | Expiry of patent term | ||
| CX01 | Expiry of patent term |
Granted publication date: 20101124 |