CN105912695A - Data information transmission and acquisition system based on Internet - Google Patents
Data information transmission and acquisition system based on Internet Download PDFInfo
- Publication number
- CN105912695A CN105912695A CN201610262043.6A CN201610262043A CN105912695A CN 105912695 A CN105912695 A CN 105912695A CN 201610262043 A CN201610262043 A CN 201610262043A CN 105912695 A CN105912695 A CN 105912695A
- Authority
- CN
- China
- Prior art keywords
- module
- data
- internet
- data information
- data processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
Abstract
The invention relates to the field of the Internet, and in particular to a data information transmission and acquisition system based on the Internet. The data information transmission and acquisition system based on the Internet comprises a data information center, an Internet module, a data transmission module, a data acquisition module, a data processing module and a database group, wherein an interconnection module is connected with the data information center through the data acquisition module; the data information center is connected with the data processing module through the data transmission module; the data processing module is connected with the database group; the data processing module comprises content extraction and batch loading and automatic classification of webpages; and the database group comprises a full text database, a relational database and an appointed folder. The data information transmission and acquisition system has the beneficial effects that combined transmission between the data acquisition module and the data transmission module is utilized to make data processing more convenient based on the Internet module, safety and reliability are good, the structure of the system is simple, and the practicality is high.
Description
Technical field
The present invention relates to internet arena, a kind of data information transfer based on the Internet and collection
System.
Background technology
Along with the development of the Internet, big data acquisition and digging technology are also evolving.Therefore, how
From the Internet, accurately, efficiently, intactly obtain mass data, just become in big data technique group
Item important technology point.
In order to obtain the data in the Internet, it is common that combine technology of creeping, it is intended to download targeted website institute
The html source code of some webpages, the most so can actually ensure to download the integrity of content.But,
In concrete practice, if directly using HTTP/HTTPS agreement to download the HTML of targeted website in client
Source code occur the problem that partial target website uses ajax technology, and by JavaScript by number
According to pouring in the page, owing to the ajax mode of different web sites is the most different, need additional analysis XHR data stream；
Website data is protected by partial target website by logging in the peripheral means such as authentication, it is therefore desirable to
Login process is carried out conversed analysis and simulation, generally goes through Code obfuscation owing to relating to the JS of encryption and decryption,
The difficulty of conversed analysis is the biggest；When directly targeted website being carried out HTTP request, it is impossible to control caching Cache,
Often duplicate the situation of download, cause the significant wastage of resource.
Summary of the invention
For above-mentioned the deficiencies in the prior art, it is an object of the invention to provide a kind of based on interconnection
The data information transfer of net and acquisition system.
For reaching above-mentioned purpose, the technical solution used in the present invention is: a kind of data based on the Internet are believed
Breath transmission and acquisition system, including data information center, the Internet module, data transmission module, data
Acquisition module, data processing module and data base's group, described interconnection module passes through data acquisition module and number
Connecting according to information centre, described data information center is connected with data processing module by data transmission module,
Described data processing module is organized with data base and is connected, and described data processing module includes contents extraction, webpage
Batch is downloaded and classification automatically, and described data base's group includes full-text database, relevant database and appointment
File.
Preferably, described data acquisition module includes controlling platform, configuration chip, display lamp, reset mould
Block, daughter board expansion slot, clock module, Ethernet interface, USB interface, solid-state memory, the first string
Mouthful and second serial, described control platform include single-chip microcomputer, bus module, intervalometer, tristate bus line bridge,
IP kernel, asynchronous receiving-transmitting transmitter, daughterboard interface, sdram controller, output/input module and RAM mould
Block.
Preferably, described single-chip microcomputer, sdram controller, daughterboard interface, asynchronous receiving-transmitting transmitter, USB
Interface, IP kernel and tristate bus line bridge are all connected with bus module.
Preferably, described configuration chip is connected with RAM module, and described display lamp and reseting module are all with defeated
Going out/input module connection, described daughter board expansion slot is connected with daughterboard interface, described first serial and second
Serial ports is all connected with asynchronous receiving-transmitting transmitter, and described Ethernet interface and solid-state memory are all and tristate bus line
Bridging connects, and described clock module is connected with intervalometer.
Owing to using technique scheme, the medicine have the advantages that based under the Internet module, utilize
Combination transmission between data acquisition module and data transmission module so that data process the most portable,
Security reliability is preferable, and system structure is simple, has the strongest practicality simultaneously.
Accompanying drawing explanation
Fig. 1 is the system block diagram of the present invention；
Fig. 2 is the data acquisition module theory diagram of the present invention.
Detailed description of the invention
Below in conjunction with accompanying drawing, embodiments of the invention are described in detail, but the present invention can be by right
Require that the multitude of different ways limiting and covering is implemented.
Such as Fig. 1 and combine shown in Fig. 2, a kind of data information transfer based on the Internet and acquisition system, bag
Include data information center, the Internet module, data transmission module, data acquisition module, data process mould
Block and data base's group, described interconnection module is connected with data information center by data acquisition module, described
Data information center is connected with data processing module by data transmission module, described data processing module and
Data base organizes connection, and described data processing module includes that contents extraction, webpage batch is downloaded and automatically classifies,
Described data base's group includes full-text database, relevant database and specified folder.
Further, described data acquisition module includes controlling platform, configuration chip, display lamp, reset
Module, daughter board expansion slot, clock module, Ethernet interface, USB interface, solid-state memory, first
Serial ports and second serial, described control platform includes single-chip microcomputer, bus module, intervalometer, tristate bus line
Bridge, IP kernel, asynchronous receiving-transmitting transmitter, daughterboard interface, sdram controller, output/input module and RAM
Module；Described single-chip microcomputer, sdram controller, daughterboard interface, asynchronous receiving-transmitting transmitter, USB interface,
IP kernel and tristate bus line bridge are all connected with bus module；Described configuration chip is connected with RAM module, described
Display lamp and reseting module are all connected with output/input module, and described daughter board expansion slot is with daughterboard interface even
Connecing, described first serial and second serial are all connected with asynchronous receiving-transmitting transmitter, described Ethernet interface and
Solid-state memory all connects with tristate bus line bridging, and described clock module is connected with intervalometer.
The medicine have the advantages that based under the Internet module, make use of data acquisition module and data to pass
Combination transmission between defeated module so that data process the most portable, and security reliability is preferable, and system is tied
Structure is simple, has the strongest practicality simultaneously.
The foregoing is only the preferred embodiments of the present invention, not thereby limit the scope of the claims of the present invention,
Every equivalent structure utilizing description of the invention and accompanying drawing content to be made or equivalence flow process conversion, or directly
Or indirectly it is used in other relevant technical fields, the most in like manner it is included in the scope of patent protection of the present invention.
Claims (2)
1. a data information transfer based on the Internet and acquisition system, it is characterised in that: include data
Information centre, the Internet module, data transmission module, data acquisition module, data processing module sum
According to storehouse group, described interconnection module is connected with data information center by data acquisition module, and described data are believed
Breath is connected with data processing module centrally through data transmission module, described data processing module and data base
Group connects, and described data processing module includes that contents extraction, webpage batch is downloaded and automatically classifies, described
Data base's group includes full-text database, relevant database and specified folder.
A kind of data information transfer based on the Internet the most according to claim 1 and acquisition system,
It is characterized in that: described data acquisition module includes controlling platform, configuration chip, display lamp, reset mould
Block, daughter board expansion slot, clock module, Ethernet interface, USB interface, solid-state memory, the first string
Mouthful and second serial, described control platform include single-chip microcomputer, bus module, intervalometer, tristate bus line bridge,
IP kernel, asynchronous receiving-transmitting transmitter, daughterboard interface, sdram controller, output/input module and RAM mould
Block；
Described single-chip microcomputer, sdram controller, daughterboard interface, asynchronous receiving-transmitting transmitter, USB interface, IP
Core and tristate bus line bridge are all connected with bus module；
Described configuration chip is connected with RAM module, described display lamp and reseting module all with input/output mould
Block connect, described daughter board expansion slot is connected with daughterboard interface, described first serial and second serial all with
Asynchronous receiving-transmitting transmitter connects, and described Ethernet interface and solid-state memory all connect with tristate bus line bridging,
Described clock module is connected with intervalometer.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610262043.6A CN105912695A (en) | 2016-04-25 | 2016-04-25 | Data information transmission and acquisition system based on Internet |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610262043.6A CN105912695A (en) | 2016-04-25 | 2016-04-25 | Data information transmission and acquisition system based on Internet |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105912695A true CN105912695A (en) | 2016-08-31 |
Family
ID=56751818
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610262043.6A Pending CN105912695A (en) | 2016-04-25 | 2016-04-25 | Data information transmission and acquisition system based on Internet |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105912695A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110048761A (en) * | 2019-04-16 | 2019-07-23 | 上海微小卫星工程中心 | One kind is towards batch production satellite data transmission ground automation high speed data processing analysis system |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101446953A (en) * | 2008-11-25 | 2009-06-03 | 北京邮电大学 | Parallel associated notice board crawler system |
| CN101499096A (en) * | 2009-03-18 | 2009-08-05 | 北京邮电大学 | Distributed reptile cluster system |
| CN101609450A (en) * | 2009-04-10 | 2009-12-23 | 南京邮电大学 | Web page classification method based on training set |
| CN101794311A (en) * | 2010-03-05 | 2010-08-04 | 南京邮电大学 | Fuzzy data mining based automatic classification method of Chinese web pages |
| CN102184262A (en) * | 2011-06-15 | 2011-09-14 | 悠易互通（北京）广告有限公司 | Web-based text classification mining system and web-based text classification mining method |
-
2016
- 2016-04-25 CN CN201610262043.6A patent/CN105912695A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101446953A (en) * | 2008-11-25 | 2009-06-03 | 北京邮电大学 | Parallel associated notice board crawler system |
| CN101499096A (en) * | 2009-03-18 | 2009-08-05 | 北京邮电大学 | Distributed reptile cluster system |
| CN101609450A (en) * | 2009-04-10 | 2009-12-23 | 南京邮电大学 | Web page classification method based on training set |
| CN101794311A (en) * | 2010-03-05 | 2010-08-04 | 南京邮电大学 | Fuzzy data mining based automatic classification method of Chinese web pages |
| CN102184262A (en) * | 2011-06-15 | 2011-09-14 | 悠易互通（北京）广告有限公司 | Web-based text classification mining system and web-based text classification mining method |
Non-Patent Citations (1)
| Title |
|---|
| 崔海逢: "《基于FPGA的GPS中频数据采集及网络传输系统设计》", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110048761A (en) * | 2019-04-16 | 2019-07-23 | 上海微小卫星工程中心 | One kind is towards batch production satellite data transmission ground automation high speed data processing analysis system |
| CN110048761B (en) * | 2019-04-16 | 2021-06-29 | 上海微小卫星工程中心 | Batch-production-oriented satellite data transmission ground automatic rapid data processing and analyzing system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Reddy et al. | Umthente uhlaba usamila: The 3rd South African national youth risk behaviour survey 2011 | |
| Lozano-Ojalvo et al. | Immune basis of allergic reactions to food. | |
| CN105279717A (en) | Internet+ intellectual property operating system and method | |
| CN108965404A (en) | Cloud net health services system and method | |
| US8539335B2 (en) | Entering data into a webpage | |
| CN105912695A (en) | Data information transmission and acquisition system based on Internet | |
| Chitiga-Mabugu et al. | Civil society organisations' participation in food security activities in South Africa | |
| Kagwanja et al. | State of the nation: South Africa 2008 | |
| Bloch et al. | Investment choices for South African education | |
| CN107895332A (en) | A kind of Software for Design seeks advice from integrated service management system | |
| Mosayebi et al. | Meta-analysis of Effectiveness of Psychological and SportInterventions on Control of META-ANALYSIS OF EFFECTIVENESS OF PSYCHOLOGICAL AND SPORT INTERVENTIONS ON CONTROL OF GLYCOSYLATED HAEMOGLOBIN IN PATIENTS WITH DIABETESin patients with diabetes | |
| Gordon et al. | Intergenerational consensus within families | |
| Dariol | BREAKING THE LINK | |
| Maluleke | African cultures and the promotion of sexual and reproductive rights | |
| Mabugu et al. | Are social grants a threat to fiscal sustainability? | |
| Bank et al. | As home spaces implode, place-making suffers | |
| April | The United States, China and mineral resource extraction: the case of the Democratic Republic of the Congo | |
| Dada et al. | Monitoring alcohol and drug abuse treatment admissions in South Africa: January-June 2013: Phase 34: SACENDU report back meetings | |
| Sithole | State democracy warming up to culture: an ambivalent integration of traditional leadership into the South African governance system, 1994-2009 | |
| Reddy et al. | What census 2011 taught us about the state of education | |
| Tassiopoulos | The world of events | |
| Roy | UNPACKING CHINA’S WHITE PAPER ON MARITIME COOPERATION UNDER BRI | |
| Sooklal et al. | Merging two developmental visons: Africa's agenda 2063 and China's one belt one road | |
| Bhana et al. | Youth work in South Africa | |
| Chan | Special Conribution to" Kokusai Mondai," No. 661 by Dr. Gerald Chan, Professor of Politics and Intern |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20160831 |
|
| WD01 | Invention patent application deemed withdrawn after publication |