CN106708918A - Network big data visualization information system - Google Patents
Network big data visualization information system Download PDFInfo
- Publication number
- CN106708918A CN106708918A CN201610531420.1A CN201610531420A CN106708918A CN 106708918 A CN106708918 A CN 106708918A CN 201610531420 A CN201610531420 A CN 201610531420A CN 106708918 A CN106708918 A CN 106708918A
- Authority
- CN
- China
- Prior art keywords
- data
- rear end
- layer
- database
- information system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/903—Querying
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/903—Querying
- G06F16/9038—Presentation of query results
Abstract
The invention discloses a network big data visualization information system, which is divided into three layers independently including a database layer, a data exchange layer and an application layer from bottom to top, wherein the database layer provides data support for the system and is used for storing original data required by the system, processed intermediate data and final display data; the data exchange layer packages a database access operation and simultaneously provides a Redis cache for caching common data, and efficiency for the application layer to read data is improved; the application layer is a WEB site and is divided into a front end and a rear end, wherein the rear end carries out underlying logic processing, database communication and exchange and data processing, and meanwhile, the rear end also provides functions including user permission, system configuration and the like; a WEB front end is used for interaction between the user and the system and carrying out visualized presentation on the data; and a front-end frame is in charge of the logic processing of front end functions, and organizing the data for providing the organized data for a visualization toolkit for carrying out rendering.
Description
Technical field
The present invention relates to network technique field, more particularly to a kind of network big data visual information system.
Background technology
In recent years developing rapidly with internet, mobile Internet and Internet of Things, the world comes into the data explosion epoch.
From the various management of commercial company and operation data, to the socialization data of individual mobile terminal, then the sea produced to internet
Amount information data, the information content for producing all the time is skyrocketed through.How relation and letter between mass data is intuitively disclosed
The feature hidden in breath, as research topic important in world wide, and then has promoted information visualization this brand-new science
The appearance in field.
Mass data visualization technique combines scientific visualization, man-machine interaction, data mining, image graphics and cognition
The theory and method of the subjects such as science, are mainly used in the abstracted information without geometric attribute, are hidden in illustration information
Knowledge.Data message and knowledge transformation are a kind of visual form by visualization, take full advantage of people to the quick knowledge of visualization model
Other natural ability.In the society that information becomes increasingly abundant, the research and application and development of visualization technique fundamentally change
The understanding mode of large complicated data, thus guide the mankind obtain it is new see clearly, and then formulate effective decision-making.
Current visualization system framework is broadly divided into Client/Server (C/S) and Browser/Server (B/S) two
The pattern of kind.C/S visualization systems are higher to the Hardware Render Capability Requirement of computer, and platform compatibility is poor, generally requires
The client software of different platform is developed, there is certain application limitation.
The content of the invention
To solve technical problem present in background technology, the present invention proposes a kind of network big data visual information system
System, using computer distribution type framework and high-efficiency network transmission technology, completes the overall frame of mass network data visualisation system
Structure is designed, and is realized based on the full stack technologies of WEB.
Therefore, the invention provides a kind of network big data visual information system, being divided into three layers, it is respectively from top to bottom
Database layer, data exchange layer and application layer；
The database layer be systems with data support, for the initial data needed for storage system, treatment after in
Between data and final display data；
The data exchange layer encapsulates the operation for accessing database, while there is provided Redis cachings, it is conventional for caching
Data, improve application layer read data efficiency；
The application layer is a WEB website, is divided into rear end and front end two parts；Wherein rear end Lower level logical is processed, counted
Communicated according to storehouse and exchanged, and data processing；Meanwhile, rear end also provides the functions such as user right, system configuration；WEB front-end is used
In user and system interaction, and visual presentation is carried out to data；Front end frame is responsible for the logical process of front-end functionality, and organizes
Data are supplied to visualization tool collection to be rendered.
According to such scheme, the data storage of the visual information system uses the non-passes of MongoDB of Key-Value forms
It is type database.
According to such scheme, using WebSocket as communications protocol, WebSocket is entering the visual information system
After TCP connection of row, follow-up data transfer is carried out based on this connection, without repeatedly sending HTTP message.
According to such scheme, the visual information system uses the data acquisition of publish/subscribe, and rear end definition is multiple
Corresponding data are subscribed in data source, front end according to demand, and when the data source of rear end changes, front end page can be real-time
Change conditions are obtained, the page that upgrades in time shows.
A kind of network big data visual information system proposed by the present invention, using computer distribution type framework and efficient net
Network transmission technology, completes the general frame design of mass network data visualisation system, and is realized based on the full stack technologies of WEB, this
System can provide comprehensive Visualization Service to mass data, contribute to it is efficient, intuitively from extracting data hiding information,
And then aid in formulating decision-making.
Brief description of the drawings
Fig. 1 is the system architecture diagram of the specific embodiment of the invention.
Fig. 2 is the MongoDB cluster topology figures of the specific embodiment of the invention.
Fig. 3 is the Data Interchange Technology schematic diagram of the specific embodiment of the invention.
Fig. 4 is the Visualization Framework schematic diagram of the specific embodiment of the invention.
Fig. 5 is the system front end Organization Chart of the specific embodiment of the invention.
Fig. 6 is the data on flows exemplary plot of the specific embodiment of the invention.
Fig. 7 is the Internet basic data statistics datagram of the specific embodiment of the invention.
Fig. 8 is the network safety event monitoring and statisticses figure of the specific embodiment of the invention.
Specific embodiment
Below, technical scheme is described in detail by specific embodiment.
Embodiment：
Reference picture 1, the present invention proposes a kind of network big data visual information system, and system is divided into three layers, from it is lower to
It is upper to be respectively database layer, data exchange layer and application layer；
Database layer is supported for systems with data, for the initial data needed for storage system, the mediant after treatment
According to this and final display data.For mass data scale, the memory space and search efficiency of unit database cannot meet
Demand, it is therefore desirable to set up data-base cluster, with lifting system performance.
Data exchange layer encapsulates the operation for accessing database, while there is provided Redis cachings, for caching conventional number
According to raising application layer reads the efficiency of data.
Application layer is a WEB website, is divided into rear end and front end two parts.Wherein rear end Lower level logical treatment, database
Communication and exchange, and data processing.Meanwhile, rear end also provides the functions such as user right, system configuration.WEB front-end is used
Family and system interaction, and visual presentation is carried out to data.Front end frame is responsible for the logical process of front-end functionality, and organizes data
It is supplied to visualization tool collection to be rendered.
Mass data storage framework for the system is as follows：
The network data of visualization system docking is numerous and jumbled, estimates in TB/PB grades of scale, to database high concurrent, efficiently reads
The requirement write is higher, and the distributed field of traditional main flow storage scheme (Oracle, MySQL etc.) and inapplicable big data quantity
Scape.Therefore, for convenience of database structure optimization and distributed expansion, data storage is non-using the MongoDB of Key-Value forms
Relevant database.Compared to traditional relevant database, it has the characteristics that：
1) unstructured data can be stored, even if among same set, data there can also be different forms, very
It is easy to extension.
2) do not support that SQL statement is operated, the query grammar of MongoDB can only be used.Current multiple programs language has official
Square chained library supports the read-write operation of MongoDB.
3) it is extending transversely easy.The horizontal fragmentation of relevant database needs to carry out the manual configuration of complexity, and MongoDB can
Auto-configuration data burst, and do load balance process.
Because system data source is numerous, and the data of separate sources often have different structures, it is necessary to according to data
Design corresponding storage model in source.MongoDB on the one hand being capable of simplified model structure, another aspect, when the new data source of increase
When, existing data model can not be changed, it is easier to extend.
For large-scale data set, memory capacity and read-write efficiency optimization are considered, establish shown in Fig. 2
MongoDB clusters.Wherein Mongod is the memory node of cluster, for data storage.ConfigDB server storages are each
The configuration information of Mongod and Mongos.Mongos stores the routing iinformation of each Mongod, application server (App
Server) communicate with.It is noted that Mongos is transparent for application server with the communication of cluster interior nodes.
Visualization Framework of the invention is as follows：
Visualization Framework is built in the WEB front-end part of system application layer.As shown in figure 4, being broadly divided into front end subscription
According to, and data visual presentation.User subscribes to data according to demand, and front end is based on HTML5 language, is added according to cognition custom
Visualization tool collection is carried to be rendered.Additionally, system provides interactive interface, data needed for customized displaying.
The technology that the present invention is utilized has：
1st, efficient data switching technology
Traditional front and back end data exchange, is all by a complete http communication.However, the header of HTTP is larger,
When the resource of request is smaller, which is extremely inefficient, while substantial amounts of bandwidth and server resource can be taken.Single visualization
The data volume of the page will not be very big, it usually needs again in the case of refresh page, is not updating display data.If front end uses
Traditional JavaScript polling modes, constantly send HTTP request, as described above, data exchange inefficiency, and largely
Expend Internet resources.
To improve data exchange efficiency, the system employs WebSocket as communications protocol.WebSocket is being carried out
After TCP connection, follow-up data transfer is carried out based on this connection, without repeatedly sending HTTP message, can be with significant increase
Data exchange efficiency.As shown in figure 3, data acquisition of the system using publish/subscribe, rear end defines multiple data sources, preceding
Corresponding data are subscribed to according to demand in end.When the data source of rear end changes, front end page can in real time obtain variation
Situation, the page that upgrades in time shows.
2nd, WEB back-end technologies
The system uses Meteor frameworks, and it is that " full stack " WEB technological frames, i.e., one framework is responsible for front end simultaneously
And rear end.
1) database schema (Schema).System Back-end data storage uses MongoDB, it set (Collection,
Table i.e. in relevant database) Schema for specific format need not be defined in fact.But Schema is defined, can be clear
Description data structure, it is easier to which data are carried out with cognitive and operation.
2) data exchange.Rear end carries out data and leads to front end using subscription (Subscribe)/issue (Publish) pattern
Letter.Rear end defines data publication interface, and according to page presentation demand, data needed for subscribing to carry out visualization and render for front end.
3) data prediction.Before data publication, data are taken out in rear end from corresponding set, if required data are not gathered
Content, but analysis or statistics, then need to pre-process data.Then corresponding issue operation is carried out again, finally
Subscribed to by front end.
4) route system.The different page of different URL correspondences, that is, correspond to different page logics and realize.Route
(Router) system is responsible for setting up logical relation between URL and the page, so as to when specific URL is accessed, perform corresponding logic,
The correct page is returned to user.
3rd, WEB front-end technology
Meteor frameworks are the platforms built on Node.js, and it is different from common MVC pattern, is not uploaded in network
Defeated HTML, server end only sends data, and client is responsible for rendering work, greatly mitigates server stress.System development, dimension
When shield or upgrading, the change of system architecture and pattern can automatically notify browser refresh page, realize " pick and send " of code
(hot push).When system is visually rendered to mass data, the data variation of server end or client all will be automatically same
Step, can significant increase data exchange, calculate and draw efficiency.System front end framework is realized as shown in Figure 5.
4th, visualization technique
The visualization technique of system application mainly includes rendering pretreatment module, mapping and display module and interaction mould
Block, wherein：
(1) pretreatment is rendered.Towards the data processing of method for visualizing, client is directed to the data for receiving and carries out visually
Change conversion (such as linear transformation, feature detection, smoothing processing etc.), to reduce the performance pressures of GPU hardware, accelerate imaging speed
Degree, lifting data mapping efficiency over the display.
(2) mapping display.Numerical matrix is converted into image, based on human vision cognition system, appropriate chart is used
Help user is more accurate, more intuitively understand data.
(3) system interaction.According to |input paramete or feedback information adjustment data visualization effect such that it is able to comprehensive,
The demonstrating data feature of various dimensions.
For large scale network data, realize that internetwork Real Data Exchangs must pay attention to data prediction work
Make, the quality of data prediction is largely fixed the comfort level of system experience.
Specifically, MongoDB clusters of the present invention are compared to relevant database, the configuration of its horizontal fragmentation is simple, and is easy to
It is extending transversely.For the network data of magnanimity scale, data storing reliability is particularly significant, it usually needs set stored copies,
Such as one master one from or a main N from mode.When host node breaks down, can be switched fast into from node and continue service, prevent
Only loss of data.The copy set configuration of MongoDB is also relatively simple, and the node in copy set backups each other, so as to reach principal and subordinate
Backup purpose.
The data source of the system is network foundation data resource, including IP address attribute knowledge base, network safety event
With traffic log etc..Fig. 6 is the example data structure of wall scroll network traffics daily record, and the average disk space taken per data is big
About 50KB.More than one hundred million of traffic log is had at present, takes disk space more than TB grades, and still constantly increasing.Except dynamic
State traffic log, other static datas take disk space and also reach more than 500GB scales.To store these data, using 10
Platform server sets up data-base cluster, and each burst is disposed using a master one by the way of.When data volume rises to now
When having memory space and cannot be competent at, can by increase burst copy set carry out it is extending transversely, with safeguards system performance.
Its effect of visualization is as follows：
Based on large scale network data, system can efficiently realize data exchange and visualization.System is to magnanimity internet
Basic data has carried out statistical analysis, and such as Global IP addresses distribution, national website and domain name is put on record situation, global WEB server
Distribution etc..As shown in fig. 7, giving effect of visualization based on national website data of putting on record, can intuitively, clearly be passed by figure
The regional development difference difference of the information contained up in big data, such as China Internet industry is big, and Middle Eastern is substantially better than
Western and the Northeast.
As shown in figure 8, establish global monitoring Early-warning Model for large-scale network security events, contribute to intuitively from
The safe potential risk that cyberspace is present is found in data.Experiment shows that system front end framework disclosure satisfy that different pieces of information
Exchange and process demand, polytype visualization tool collection effectively supports the displaying requirement of multi-source mass data.
Based on above-mentioned, the present embodiment is to the resource-sharing of current WEB visualization systems, interconnection is rebuild and exchange capacity is not enough
The problems such as, using distributed storage architecture and high-efficiency network Data Interchange Technology, devise visualization system overall traffic and
Functional framework, and a set of visualization system based on network big data is realized based on the full stack technologies of WEB.Result proves that this is
System is realized to network data storage, calculating and effective utilization of rendering resource, it is possible to which data are provided with comprehensively visualization
Service, contribute to it is efficient, useful information is intuitively extracted from large-scale data, and then aid in formulating decision-making.
The above, the only present invention preferably specific embodiment, but protection scope of the present invention is not limited thereto,
Any one skilled in the art the invention discloses technical scope in, technology according to the present invention scheme and its
Inventive concept is subject to equivalent or change, should all be included within the scope of the present invention.
Claims (4)
1. a kind of network big data visual information system, it is characterised in that be divided into three layers, from top to bottom respectively database
Layer, data exchange layer and application layer；
The database layer is supported for systems with data, for the initial data needed for storage system, the mediant after treatment
According to this and final display data；
The data exchange layer encapsulates the operation for accessing database, while there is provided Redis cachings, for caching conventional number
According to raising application layer reads the efficiency of data；
The application layer is a WEB website, is divided into rear end and front end two parts；Wherein rear end Lower level logical treatment, database
Communication and exchange, and data processing；Meanwhile, rear end also provides the functions such as user right, system configuration；WEB front-end is used
Family and system interaction, and visual presentation is carried out to data；Front end frame is responsible for the logical process of front-end functionality, and organizes data
It is supplied to visualization tool collection to be rendered.
2. a kind of network big data visual information system as claimed in claim 1, it is characterised in that the visual information system
The data storage of system uses the MongoDB non-relational databases of Key-Value forms.
3. a kind of network big data visual information system as claimed in claim 1, it is characterised in that the visual information system
Using WebSocket as communications protocol, WebSocket is carried out follow-up system after a TCP connection is carried out based on this connection
Data transfer, without repeatedly sending HTTP message.
4. a kind of network big data visual information system as claimed in claim 1, it is characterised in that the visual information system
System uses the data acquisition of publish/subscribe, and corresponding data are subscribed in the multiple data sources of rear end definition, front end according to demand,
When the data source of rear end changes, front end page can in real time obtain change conditions, and the page that upgrades in time shows.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610531420.1A CN106708918A (en) | 2016-06-29 | 2016-06-29 | Network big data visualization information system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610531420.1A CN106708918A (en) | 2016-06-29 | 2016-06-29 | Network big data visualization information system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106708918A true CN106708918A (en) | 2017-05-24 |
Family
ID=58940749
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610531420.1A Pending CN106708918A (en) | 2016-06-29 | 2016-06-29 | Network big data visualization information system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106708918A (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107391686A (en) * | 2017-07-24 | 2017-11-24 | 威创软件南京有限公司 | A kind of visual configuration data collecting system implementation method |
| CN107888429A (en) * | 2017-12-06 | 2018-04-06 | 北京连琪科技有限公司 | Block chain running status method for visualizing, device and browser |
| CN108134819A (en) * | 2017-12-06 | 2018-06-08 | 北京连琪科技有限公司 | Block chain operating status is collected and transfer approach, device and server |
| CN108509566A (en) * | 2018-03-26 | 2018-09-07 | 国家电网公司客户服务中心 | One kind is based on 95598 data publication service operation system network topology method for visualizing on cloud |
| CN108563666A (en) * | 2018-01-05 | 2018-09-21 | 成都兴政电子政务运营服务有限公司 | A kind of data visualization processing system and method based on big data technology |
| CN108646691A (en) * | 2018-07-09 | 2018-10-12 | 鹤壁市煤化机械有限责任公司 | Oscillating feeder online monitoring system |
| CN109948049A (en) * | 2019-01-18 | 2019-06-28 | 杭州志远科技有限公司 | A kind of network big data method for visualizing based on materialization caching |
| CN111444230A (en) * | 2019-01-17 | 2020-07-24 | 苏州黑牛新媒体有限公司 | Data visualization analysis method based on big data platform |
| CN111881204A (en) * | 2020-07-24 | 2020-11-03 | 海南中金德航科技股份有限公司 | Big data visualization platform |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101216926A (en) * | 2008-01-09 | 2008-07-09 | 深圳市海力特科技有限责任公司 | An urban emergency commanding operation system and the corresponding implementation method |
| CN104156465A (en) * | 2014-08-22 | 2014-11-19 | 金石易诚（北京）科技有限公司 | Real-time webpage synchronization and background distributed data storage system |
-
2016
- 2016-06-29 CN CN201610531420.1A patent/CN106708918A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101216926A (en) * | 2008-01-09 | 2008-07-09 | 深圳市海力特科技有限责任公司 | An urban emergency commanding operation system and the corresponding implementation method |
| CN104156465A (en) * | 2014-08-22 | 2014-11-19 | 金石易诚（北京）科技有限公司 | Real-time webpage synchronization and background distributed data storage system |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107391686A (en) * | 2017-07-24 | 2017-11-24 | 威创软件南京有限公司 | A kind of visual configuration data collecting system implementation method |
| CN107888429A (en) * | 2017-12-06 | 2018-04-06 | 北京连琪科技有限公司 | Block chain running status method for visualizing, device and browser |
| CN108134819A (en) * | 2017-12-06 | 2018-06-08 | 北京连琪科技有限公司 | Block chain operating status is collected and transfer approach, device and server |
| CN108563666A (en) * | 2018-01-05 | 2018-09-21 | 成都兴政电子政务运营服务有限公司 | A kind of data visualization processing system and method based on big data technology |
| CN108563666B (en) * | 2018-01-05 | 2022-04-05 | 四川兴政信息技术有限公司 | Data visualization processing system and method based on big data technology |
| CN108509566A (en) * | 2018-03-26 | 2018-09-07 | 国家电网公司客户服务中心 | One kind is based on 95598 data publication service operation system network topology method for visualizing on cloud |
| CN108646691A (en) * | 2018-07-09 | 2018-10-12 | 鹤壁市煤化机械有限责任公司 | Oscillating feeder online monitoring system |
| CN111444230A (en) * | 2019-01-17 | 2020-07-24 | 苏州黑牛新媒体有限公司 | Data visualization analysis method based on big data platform |
| CN109948049A (en) * | 2019-01-18 | 2019-06-28 | 杭州志远科技有限公司 | A kind of network big data method for visualizing based on materialization caching |
| CN109948049B (en) * | 2019-01-18 | 2021-04-13 | 杭州志远科技有限公司 | Network big data visualization method based on materialized cache |
| CN111881204A (en) * | 2020-07-24 | 2020-11-03 | 海南中金德航科技股份有限公司 | Big data visualization platform |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106708918A (en) | Network big data visualization information system | |
| CN111542813B (en) | Object model using heterogeneous data to facilitate building data visualizations | |
| Kraska | Finding the needle in the big data systems haystack | |
| CN102012906B (en) | Three-dimensional scene management platform based on SaaS architecture and editing and browsing method | |
| CN106372194B (en) | Method and system for presenting search results | |
| CN107291806A (en) | A kind of Data View copy alternative manner in Web visible environments | |
| CN104216989A (en) | Method for storing transmission line integrated data based on HBase | |
| CN110928740A (en) | Centralized visualization method and system for operation and maintenance data of cloud computing center | |
| CN111639114A (en) | Distributed data fusion management system based on Internet of things platform | |
| CN107302573A (en) | A kind of information-pushing method, device, electronic equipment and storage medium | |
| CN110020273A (en) | For generating the method, apparatus and system of thermodynamic chart | |
| CN104317842A (en) | Intelligent data extraction and thematic map generation engine based on geographic information technology | |
| CN103823818A (en) | Book system on basis of virtual reality | |
| CN103823549A (en) | Somatosensory movie playing system | |
| CN111352951A (en) | Data export method, device and system | |
| CN107679151A (en) | A kind of data processing method based on ELA big data flight deck systems | |
| CN105930354B (en) | Storage model conversion method and device | |
| CN106599313A (en) | Visual data cognition method | |
| CN113918669A (en) | Device and method for realizing natural resource homeland space planning one-map system | |
| Wu et al. | Research on data sharing architecture for ecological monitoring using Iot streaming data | |
| CN109543087A (en) | A kind of construction method and application method of international three pole data interoperation automotive engine system | |
| CN109164764A (en) | Workshop wholegrain degree three-dimensional visualization monitoring system and construction method | |
| CN107515913A (en) | A kind of multivariate data model integrated construction method and its virtual interactive interface system | |
| Bruhwiler et al. | Iris: Amortized, resource efficient visualizations of voluminous spatiotemporal datasets | |
| CN109033447A (en) | A kind of facial recognition data visualization system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20170524 |