CN108363756A - A kind of intelligent transportation big data processing system - Google Patents
A kind of intelligent transportation big data processing system Download PDFInfo
- Publication number
- CN108363756A CN108363756A CN201810098106.8A CN201810098106A CN108363756A CN 108363756 A CN108363756 A CN 108363756A CN 201810098106 A CN201810098106 A CN 201810098106A CN 108363756 A CN108363756 A CN 108363756A
- Authority
- CN
- China
- Prior art keywords
- big data
- data
- analysis
- processing system
- data processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
- G06F16/285—Clustering or classification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2465—Query processing support for facilitating data mining operations in structured databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/951—Indexing; Web crawling techniques
-
- G—PHYSICS
- G08—SIGNALLING
- G08G—TRAFFIC CONTROL SYSTEMS
- G08G1/00—Traffic control systems for road vehicles
- G08G1/01—Detecting movement of traffic to be counted or controlled
- G08G1/0104—Measuring and analyzing of parameters relative to traffic conditions
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/55—Push-based network services
Abstract
An embodiment of the present invention provides a kind of intelligent transportation big data processing system, the system comprises：Big data platform, for storing the big data for receiving and having acquired；Big data preprocessing module, for being pre-processed to the big data acquired；The pretreatment is for realizing load balancing, resource virtualizing and Distributed Storage management；Intelligent analysis module for carrying out the analysis of the big data according to the instruction received, and comprehensively utilizes the data of big data preprocessing module offer, carries out event causality analysis；Data processing system module passes to IN service analysis module for generating data analysis task and receives the result of the intelligent analysis module return；The analysis assessment result of company information, all kinds of details data hierarchy grades are subjected to synthesis by the visualization technique that can be interacted, showed in real time.Using the embodiment of the present invention, solve the problems, such as that traditional data processing method real-time, high efficiency and interactivity are poor.
Description
Technical field
The present invention relates to electronic technology field more particularly to a kind of intelligent transportation big data processing systems.
Background technology
With society's industrialization, the continuous improvement of the level of IT application, nowadays data, which have replaced, is calculated as information calculating
Center, cloud computing, big data are becoming a kind of trend and trend, including memory capacity, availability, I/O performances, data safety
All various aspects such as property, scalability.Big data is the very huge and complicated data set of scale.Big data has 4V：Volume is (big
Amount), data volume increases continuously and healthily；Velocity (high speed), data I/O speed are faster；Variety (various), data type
With source diversification；Value (value), there are the usable values of various aspects.How from the extracting data of magnanimity, thought
The knowledge or interested information wanted, this is the requirement made good use of big data, and then preferably serve social development.Therefore,
Data digging method comes into being.
Data mining is born in as a subject in the 1980s, being exactly from large amount of complex from the point of view of technology
, obtain information implicit, that people do not realize in advance, having potential value in irregular, random, fuzzy data
With the process of knowledge.In big data application field, user group can be often divided into according to the various actions feature of user
If Ganlei, in order to carry out accurate formula, personalized service for the feature of user group.Cluster is divided to user group
A kind of mode.Cluster is so that the object in same class is had very high similarity the classified process of data object, and make
Object height in inhomogeneity is different.Distinctiveness ratio is measured usually using distance.
But the effect that user group is divided for user behavior characteristics in cluster operation largely according to
The quality in basic data, the existing user group based on clustering algorithm is relied to divide and often can not reflect user's well
There is cluster inaccuracy in behavioural characteristic, it is difficult to carry out accurate formula, personalized clothes to user group using cluster result
Business.
Invention content
The embodiment of the present invention is designed to provide a kind of intelligent transportation big data processing system, solves at traditional data
Reason method real-time, high efficiency and the poor problem of interactivity, help user to perceive conditions of the enterprise in real time, to improve enterprise
The efficiency of management and business processing are horizontal.
In order to achieve the above object, the embodiment of the invention discloses a kind of intelligent transportation big data processing system, the systems
System includes：
Traffic big data for storing city is simultaneously managed traffic big data, and the traffic big data includes urban road
Information data, city automobile management information data, urban meteorological data, city automobile real time positioning data and urban transportation prison
Control video data；
Big data preprocessing module, for the traffic big data into pre-processing；The pretreatment is for realizing load
Balanced, resource virtualizing and Distributed Storage management；
Intelligent analysis module, the analysis for carrying out the big data according to the instruction received, and it is pre- to comprehensively utilize big data
The data that processing module provides carry out event causality analysis；
Data processing system module passes to IN service analysis module and receives the intelligence for generating data analysis task
The result that analysis module returns；By the visualization technique that can interact by the analysis assessment result of company information, all kinds of details numbers
Synthesis is carried out according to hierarchical, is showed in real time, and provides operation interface to the user.
Optionally, the operation interface includes at least one or more in following functions：With Visual Chart, analysis
Report, content retrieval and message push/subscription.
Optionally, the big data preprocessing module
By the big data acquired described in distributed system architecture Hadoop storage, and for realizing load balancing,
Resource virtualizing, Distributed Storage management and application programming interface api interface function.
Optionally, the analysis module, for realizing extemporaneous inquiry/combination condition query, multidimensional OLAP, KPI indexs,
The analytic functions such as MDX inquiries
Optionally, the analysis module, for realizing data mining capabilities such as classification, cluster, correlation rules, and flexible ginseng
Number configuration feature；
Optionally, index evaluation module assesses in real time for carrying out real-time company's situation according to preset data information
Optionally, the preset data information includes：Manpower, finance, substance and business.
Optionally, the system client that Javaweb technologies have built B/S patterns may be used in described information display module,
Single-sign-on control is realized, provides displaying, operation interface to the user；Abundant figure is created using the ExtJS frames increased income
And chart, business information analysis assessment result, all kinds of details data hierarchy grades are subjected to synthesis, showed in real time；System users
End has the function of Visual Chart, analysis report, content retrieval, message push/subscription etc., can moved by browser execution
Moved end, the ends PC.
Intelligent transportation big data processing system provided in an embodiment of the present invention, compared with prior art, the present invention has as follows
Advantage：
1, processing speed is fast：System architecture scheme carries out rational management, Ke Yichong using big data technology to calculating, store tasks
The operational capability of each clustered node in system is waved in distribution；It, can be conveniently by addition cluster section when business demand increases
Point comes expansion system scale, lifting system performance.
2, better user experience：System supports multiple terminal operation, supports the real-time visual of feelings indexs in school at different levels, provides
The interactive mode of simple, intuitive；
3, flexibility ratio is high：It can flexibly be created according to the actual conditions of this enterprise, Allocation Analysis model；System is set using layering
Meter is easy to deployment implementation, upgrade maintenance.
Description of the drawings
In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below
There is attached drawing needed in technology description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this
Some embodiments of invention for those of ordinary skill in the art without creative efforts, can be with
Obtain other attached drawings according to these attached drawings.
Fig. 1 is the first structural schematic diagram of intelligent transportation big data processing system provided in an embodiment of the present invention.
Fig. 2 is second of structural schematic diagram of intelligent transportation big data processing system provided in an embodiment of the present invention.
Fig. 3 is the third structural schematic diagram of intelligent transportation big data processing system provided in an embodiment of the present invention.
Specific implementation mode
Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, complete
Site preparation describes, it is clear that described embodiments are only a part of the embodiments of the present invention, instead of all the embodiments.It is based on
Embodiment in the present invention, it is obtained by those of ordinary skill in the art without making creative efforts every other
Embodiment shall fall within the protection scope of the present invention.
As shown in Figure 1 to Figure 3, it is the structural representation of intelligent transportation big data processing system provided in an embodiment of the present invention
Figure, is described in detail separately below.
Big data platform 11, traffic big data for storing city are simultaneously managed traffic big data, the traffic
Big data includes that urban road information data, city automobile management information data, urban meteorological data, city automobile position in real time
Data and urban transportation monitor video data；
Big data preprocessing module 12, for the traffic big data into pre-processing；The pretreatment is for realizing negative
Carry balanced, resource virtualizing and Distributed Storage management；
Intelligent analysis module 13 for carrying out the analysis of the big data according to the instruction received, and comprehensively utilizes big data
The data that preprocessing module provides carry out event causality analysis；
Data processing system module 14 passes to IN service analysis module and receives the intelligence for generating data analysis task
The result that energy analysis module returns；By the visualization technique that can interact by the analysis assessment result of company information, all kinds of details
Data hierarchy grade carries out synthesis, shows in real time, and provides operation interface to the user.
In the present embodiment, entire analysis platform is to realize cloud platform, cluster hardware configuration based on the Hadoop framework increased income
It is as follows：The memory of the GB of the CPU, 64 or 128 of 16 core, 32 thread, multiple default turns be directly connected to by the mainboard controller of CPU
The hard disk (always storing up to 24TB) of speed, and cluster is built using gigabit Ethernet.Wherein, the quantity of hard disk and rotating speed can bases
It needs to be configured, if quantity is 20, rotating speed is 3600r/s etc..
Wherein, there are four types of basic task roles for Hadoop clusters：Name node (including alternative name node), work shadow
Node, task execution node and back end.Name node is responsible for coordinating the data storage on cluster；Job trace node is responsible for
Coordination data handles task；Task execution node is responsible for carrying out the tasks such as data acquisition, data processing；Back end is responsible for storage
Data.Most node needs while being used as back end and task execution node in cluster.
On the basis of Hadoop clusters, the support handled distributed parallel task is realized by Map/Reduce.
Map/Reduce is a programming model for being used for big data quantity parallel computation, while being also a kind of efficient task scheduling mould
Type, one big task is divided into many more fine-grained subtasks by it, by carrying out subtask between idle processing node
Scheduling, the node for avoiding processing speed slow extend the deadline of entire task.
In the present invention, entire intellectualized analysis platform includes three parts, such as big data preprocessing module 11, intellectual analysis mould
Block 12 and data processing system modules 14；Wherein, big data warehouse has been built in big data preprocessing module 11, for storing
Crucial initial data in the data source of ETL process abstractions builds cube (Cube) on the basis of big data warehouse,
Data are provided and are supported for network analysis, displaying.
Big data preprocessing module 12 is additionally operable to realize by cloud platform management module equal to the load of bottom layer node equipment
The functions such as weighing apparatus, resource virtualizing, Distributed Storage management, fault-tolerant strategy management and offer api interface, realize big data
Processing and management.
Above-mentioned data source is each business department of enterprise independently all operation system and database, including HR Office's number
According to, Finance Department's data, research and development department's data, Finance Department's data, market department's data, data at assessment, interconnection wet end data and comprehensive
Close management data；Wherein, above-mentioned part may segment many small departments, if market department may further include secretary's group, market group
Deng data at this time are included in market department.Big data preprocessing module is deposited by distributed system architecture Hadoop
The big data acquired described in storage, and for realizing load balancing, resource virtualizing, Distributed Storage management and application
Program Interfaces api interface function.
The intelligent analysis module 13, for realizing extemporaneous inquiry/combination condition query, multidimensional OLAP, KPI indexs, MDX
The analytic functions such as inquiry, while realizing the data mining capabilities such as classification, cluster, correlation rule, and flexible parameter configuration work(
Energy；Index evaluation module 15 is assessed in real time for carrying out real-time company's situation according to preset data information；The present count it is believed that
Breath includes：Manpower, finance, substance and business.
Above-mentioned big data warehouse is based on HDFS and Hive and realizes, has concentrated enterprise each mutually using distributed storage mode
Mass data in independent operation system provides data for cube.Further, the data in big data warehouse with
Dimension and the storage of the form of true table, dimension here are the attributes of data, indicate that the angle of analysis data, type have general dimension
Degree, time dimension and gradual change dimension；True table is the main table for storing the data to be analyzed, only includes major key, external key and measurement
Value.
Operation interface includes at least one or more in following functions：With Visual Chart, analysis report, content inspection
Rope and message push/subscription.
In the embodiment of the present invention, data processing system module 14 has built the system visitor of B/S patterns using Javaweb technologies
Family end realizes single-sign-on control, provides displaying, operation interface to the user；It is created using the ExtJS frames increased income abundant
Business information analysis assessment result, all kinds of details data hierarchy grades are carried out synthesis, showed in real time by figure and chart；System
Client has the function of Visual Chart, analysis report, content retrieval, message push/subscription etc., can pass through browser execution
At mobile terminal, the ends PC.
It should be noted that herein, relational terms such as first and second and the like are used merely to a reality
Body or operation are distinguished with another entity or operation, are deposited without necessarily requiring or implying between these entities or operation
In any actual relationship or order or sequence.Moreover, the terms "include", "comprise" or its any other variant are intended to
Non-exclusive inclusion, so that the process, method, article or equipment including a series of elements is not only wanted including those
Element, but also include other elements that are not explicitly listed, or further include for this process, method, article or equipment
Intrinsic element.In the absence of more restrictions, the element limited by sentence "including a ...".The above is only
For presently preferred embodiments of the present invention, it is not intended to limit the scope of the present invention.It is all in the spirit and principles in the present invention
Within any modification, equivalent replacement, improvement and so on, be included within the scope of protection of the present invention.
Claims (8)
1. a kind of intelligent transportation big data processing system, which is characterized in that the system comprises：
Big data platform, traffic big data for storing city are simultaneously managed traffic big data, the traffic big data
Including urban road information data, city automobile management information data, urban meteorological data, city automobile real time positioning data with
And urban transportation monitor video data；
Big data preprocessing module, for the traffic big data into pre-processing；The pretreatment is for realizing load
Balanced, resource virtualizing and Distributed Storage management；
Intelligent analysis module, the analysis for carrying out the big data according to the instruction received, and it is pre- to comprehensively utilize big data
The data that processing module provides carry out event causality analysis；
Data processing system module passes to IN service analysis module and receives the intelligence for generating data analysis task
The result that analysis module returns；By the visualization technique that can interact by the analysis assessment result of company information, all kinds of details numbers
Synthesis is carried out according to hierarchical, is showed in real time, and provides operation interface to the user.
2. a kind of intelligent transportation big data processing system according to claim 1, which is characterized in that the operation interface is extremely
Include one or more in following functions less：With Visual Chart, analysis report, content retrieval and message push/subscription.
3. a kind of intelligent transportation big data processing system according to claim 1, which is characterized in that the big data is located in advance
Module is managed, for the big data by having been acquired described in the Hadoop storages of distributed system architecture, and for realizing negative
Carry equilibrium, resource virtualizing, Distributed Storage management and application programming interface api interface function.
4. a kind of intelligent transportation big data processing system according to claim 1, which is characterized in that the intellectual analysis mould
Block, for realizing analytic functions such as extemporaneous inquiry/combination condition query, multidimensional OLAP, KPI indexs, MDX inquiries.
5. a kind of intelligent transportation big data processing system according to claim 1, which is characterized in that realize classification, cluster,
The data mining capabilities such as correlation rule, and flexible parameter configuration function.
6. a kind of intelligent transportation big data processing system according to claim 1, which is characterized in that the system is also wrapped
It includes：Index evaluation module is assessed in real time for carrying out real-time company's situation according to preset data information.
7. a kind of intelligent transportation big data processing system according to claim 1, which is characterized in that the present count it is believed that
Breath includes：Manpower, finance, substance and business.
8. a kind of intelligent transportation big data processing system according to claim 1, which is characterized in that described information shows mould
The system client that Javaweb technologies have built B/S patterns may be used in block, realizes single-sign-on control, provides to the user
Displaying, operation interface；Abundant figure and chart are created using the ExtJS frames increased income, by business information analysis assessment result,
All kinds of details data hierarchy grades carry out synthesis, show in real time；System client has Visual Chart, analysis report, content
The functions such as retrieval, message push/subscription, can be by browser execution at mobile terminal, the ends PC.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810098106.8A CN108363756A (en) | 2018-01-31 | 2018-01-31 | A kind of intelligent transportation big data processing system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810098106.8A CN108363756A (en) | 2018-01-31 | 2018-01-31 | A kind of intelligent transportation big data processing system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108363756A true CN108363756A (en) | 2018-08-03 |
Family
ID=63007827
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810098106.8A Withdrawn CN108363756A (en) | 2018-01-31 | 2018-01-31 | A kind of intelligent transportation big data processing system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108363756A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109032144A (en) * | 2018-08-21 | 2018-12-18 | 武汉理工大学 | A kind of driver's personalized driving characteristic learning system based on big data |
| CN109410588A (en) * | 2018-12-20 | 2019-03-01 | 湖南晖龙集团股份有限公司 | A kind of traffic accident evolution analysis method based on traffic big data |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104915793A (en) * | 2015-06-30 | 2015-09-16 | 北京西塔网络科技股份有限公司 | Public information intelligent analysis platform based on big data analysis and mining |
| CN105022778A (en) * | 2015-05-04 | 2015-11-04 | 南京智行信息科技有限公司 | Intelligent traffic management big data analysis system |
| CN106504522A (en) * | 2016-12-02 | 2017-03-15 | 深圳市智慧交通有限公司 | A kind of wisdom traffic management system based on big data |
| CN106846815A (en) * | 2017-03-29 | 2017-06-13 | 深圳万智联合科技有限公司 | A kind of efficient intelligent traffic administration system big data analysis system |
| CN107590181A (en) * | 2017-08-01 | 2018-01-16 | 佛山市深研信息技术有限公司 | A kind of intelligent analysis system of big data |
-
2018
- 2018-01-31 CN CN201810098106.8A patent/CN108363756A/en not_active Withdrawn
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105022778A (en) * | 2015-05-04 | 2015-11-04 | 南京智行信息科技有限公司 | Intelligent traffic management big data analysis system |
| CN104915793A (en) * | 2015-06-30 | 2015-09-16 | 北京西塔网络科技股份有限公司 | Public information intelligent analysis platform based on big data analysis and mining |
| CN106504522A (en) * | 2016-12-02 | 2017-03-15 | 深圳市智慧交通有限公司 | A kind of wisdom traffic management system based on big data |
| CN106846815A (en) * | 2017-03-29 | 2017-06-13 | 深圳万智联合科技有限公司 | A kind of efficient intelligent traffic administration system big data analysis system |
| CN107590181A (en) * | 2017-08-01 | 2018-01-16 | 佛山市深研信息技术有限公司 | A kind of intelligent analysis system of big data |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109032144A (en) * | 2018-08-21 | 2018-12-18 | 武汉理工大学 | A kind of driver's personalized driving characteristic learning system based on big data |
| CN109410588A (en) * | 2018-12-20 | 2019-03-01 | 湖南晖龙集团股份有限公司 | A kind of traffic accident evolution analysis method based on traffic big data |
| CN109410588B (en) * | 2018-12-20 | 2022-03-15 | 湖南晖龙集团股份有限公司 | Traffic accident evolution analysis method based on traffic big data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107590181A (en) | A kind of intelligent analysis system of big data | |
| CN110199273B (en) | System and method for loading, aggregating and bulk computing in one scan in a multidimensional database environment | |
| CN109766374B (en) | Credit joint supervision platform | |
| US20200073876A1 (en) | Scalable indexing architecture | |
| US20190005104A1 (en) | Systems and methods for interest-driven data visualization systems utilizing visualization image data and trellised visualizations | |
| CN111523003A (en) | Data application method and platform with time sequence dynamic map as core | |
| CN104573071A (en) | Intelligent school situation analysis system and method based on megadata technology | |
| US9934299B2 (en) | Systems and methods for interest-driven data visualization systems utilizing visualization image data and trellised visualizations | |
| CN104915793A (en) | Public information intelligent analysis platform based on big data analysis and mining | |
| US20150363465A1 (en) | Method and system for processing queries over datasets stored using hierarchical data structures | |
| CN105139281A (en) | Method and system for processing big data of electric power marketing | |
| CN108228775A (en) | Land and resources datas management system based on cloud platform | |
| Chen et al. | Development and application of big data platform for garlic industry chain | |
| CN107291539B (en) | Cluster program scheduler method based on resource significance level | |
| CN114416855A (en) | Visualization platform and method based on electric power big data | |
| CN108399208A (en) | A kind of information display system of big data | |
| CN109033113A (en) | The management method and device of data warehouse and Data Mart | |
| CN108363756A (en) | A kind of intelligent transportation big data processing system | |
| Hu | E-commerce big data computing platform system based on distributed computing logistics information | |
| CN111209323A (en) | Spatial geographic information big data processing system | |
| CN108304549A (en) | A kind of big data Intelligent processing system | |
| CN106575296A (en) | Dynamic N-dimensional cubes for hosted analytics | |
| CN108306916A (en) | Big data multi-internet integration scientific research all-in-one machine stage apparatus | |
| CN110489732A (en) | Method for processing report data and equipment | |
| Gorawski et al. | Materialized ar-tree in distributed spatial data warehouse |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20180803 |