CN110532283A - A kind of smart city big data processing system based on Hadoop aggregated structure - Google Patents
A kind of smart city big data processing system based on Hadoop aggregated structure Download PDFInfo
- Publication number
- CN110532283A CN110532283A CN201910827032.1A CN201910827032A CN110532283A CN 110532283 A CN110532283 A CN 110532283A CN 201910827032 A CN201910827032 A CN 201910827032A CN 110532283 A CN110532283 A CN 110532283A
- Authority
- CN
- China
- Prior art keywords
- data
- interface
- hadoop
- spark
- aggregated structure
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/113—Details of archiving
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/26—Visual data mining; Browsing structured data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Abstract
The invention belongs to resource information processing technical fields, disclose a kind of smart city big data processing system based on Hadoop aggregated structure, acquisition layer is for acquiring data；Inclusion layer is used to carry out the data of acquisition aggregation processing, realizes and carries out unified loading, classification, processing and storage to the data of multiple data sources；Application layer is used to model and utilize mobile device, plate, computer or other terminal devices that each functional module is presented using algorithms of different based on function difference.Each Regional Population Floating can be predicted in the present invention, inquires into reasonable distribution social resources and the strategy of sustainable development, provides data supporting for government decision, has preferable execution efficiency and scalability；The present invention can preferably be suitable for data mining and machine learning, the speed of service is quick using SPARK open source parallel architecture design.
Description
Technical field
The invention belongs to resource information processing technical field more particularly to a kind of wisdom cities based on Hadoop aggregated structure
City's big data processing system.
Background technique
Currently, the prior art commonly used in the trade is such that
With the development of the times, mobile phone has become our indispensable interactive tools.It is fast with mobile terminal quantity
Speed increases and the reinforcement of processing capacity, and each major company, which all recognizes, occupies mobile terminal market, just occupies huge client,
A large amount of information of mobile user has been grasped, the main trend of the following mobile Internet is occupied.And 3G, 4G network it is fast-developing with
Also bring a large amount of completely new research and using chance.Such as the service based on geographical location information, technology of Internet of things
In conjunction with and mobile social networking related data excavate etc..
Trade center and aggregation center of the city as the mankind, are the products that human economic society develops to certain phase.
The appearance in city is that human society steps into the mark in civilization and the advanced form of the gregarious life of the mankind.Urbanization process
Quickening, so that city is had been assigned unprecedented economic, politics and technology right, generation has inevitably been shifted onto city
The center of boundary's stage, plays leading role.At the same time, city be also faced with environmental pollution, traffic jam, energy shortages,
The challenge of housing shortage, unemployment, disease etc..Under new environment, problems brought by urban development how are solved, it is real
Existing sustainable development becomes the important proposition of urban planning and construction.Under this overall situation, the theory of smart city is come into being.
Smart city covers the every aspect in city, including medical treatment, education, traffic, community, environmental protection, agricultural etc., it is based on Internet of Things
On the basis of net, cloud computing, big data, the urban informationization that generation information technology is applied to the various aspects in city is advanced
Form.
The high speed development of information technology accelerates urbanization process in the process, and the sharp increase of urban population also increases
City management difficulty, such as traffic pressure, employment pressure etc..Analysis urban population flow behavior facilitates reasonable distribution society money
Source, successfully manage traffic pressure, safeguard social king's peace etc..Traditional manual analysis method, such as questionnaire survey, access of having an informal discussion
Deng with high costs and inefficient.The continuous development of smart phone is bringing the same of great convenience for people's daily life with popularizing
When, generated mobile phone user's signaling data provides possibility for effectively analysis urban population flow behavior.However, magnanimity, low
The user data of matter brings lot of challenges to query analysis work.Since the economic development of ground section is unbalanced, urban inner is each
The function allocation in region is had nothing in common with each other, and causes urban inner population that can largely flow.It is limited to the factors such as geographical and social activity, people
Behavior often show regularity, be exactly periodic location transition of the people in place of working and residence.
In conclusion problem of the existing technology is:
(1) data in mobile phone analysis is different in the movable spatial and temporal distributions of different type, the method for activity intensity, does not have
Corresponding cross-section study.
(2) crowd's flow-data is influenced by factors such as weather, festivals or holidays, the coast is clear situations, according only to life
The comparison of regional population's number living has been difficult to meet demand
(3) involuntary offer data in mobile phone can not record User Activity purpose, and employment, trip can not be distinguished directly from data
The Activity Types such as rest, inhabitation.
(4) existing wisdom platform can not accurately analyze movement of population；Their location festivals or holidays celebration is numerous, in survey region
Crowd's flowing is irregular, and the regional analysis of normality is difficult to carry out.
(5) magnanimity, low-quality user data to query analysis work bring lot of challenges；Base station data of fixed place and time
Collect to crowd's flow analysis in region make data update not in time, system response it is slow, can not be to emergency event or section
Crowd's flowing of holiday is reasonably handled, and in the prior art, is not based on mobile subscriber's signaling data and is predicted each area
Domain movement of population, cannot effectively analyze social resources can distribute data, and without using hadoop framework, cannot be effectively reduced
Development cost；Not using SPARK open source parallel architecture, speed of service speed in data mining and machine learning is caused.
Solve the difficulty of above-mentioned technical problem:
Most important difficulty is the analysis mode of city space and the analysis of metropolitan statical area, how to pass through network service
Qu Zhong calculates coverage size of the service area under certain restrictions by the setting to parameter, obtains the service model of site
It encloses and accessibility.The setting into Quzhou area to impedance data is combined, can be using any cost nature as impedance, and be somebody's turn to do
Impedance needs are added up in determining service area.Constantly setting impedance value 1min, 2min, 5min, 10mi you, and sequentially tie
It calculates the influence situation of site in different impedances and carries out Comparative result after analyzing result, then most reasonable resistance is found out by dichotomy
Anti- value.In the concentric circles for passing through multiple time faces, while the coverage in distance 10km, 20km, 50km, 100km is searched, from
And it establishes reasonable analyzed area and divides.
Solve the meaning of above-mentioned technical problem:
The research of hot spot region Crowds Distribute: population collection is presented in a manner of thermodynamic chart, and modes of warning is provided.Mainly grind
Exploration is studied carefully under emergency case and crowd's temperature in occasion region, provides data supporting in public safety field.
Real-time Road wagon flow statistical research: being calculated the congestion level of every road by algorithm, and be converted into speed, can be real
Existing different time granularity, provides data supporting for field of traffic.
Tourist attractions personnel research: realizing scenic spot 24 hours and monitor in real time, analyze the distribution situation of inflow and outflow personnel,
Big data analysis data are provided for tourism industry.
Emergency worker search and rescue field research: to population be missing user carry out trace analysis processing, for emergency worker search and rescue
Data supporting and positioning service are provided.
The research of population from other places: distribution and the mobility status of thoroughfare user is carried out in analysis other places, provides for public security investigation certain
Data supporting.
The sustainable development in city and the emphasis to develop in a healthy way as present urban construction, therefore, at this stage to wisdom
The construction in city is studied, and has important theory significance and practice significance.
Summary of the invention
In view of the problems of the existing technology, the present invention provides a kind of smart city based on Hadoop aggregated structure is big
Data processing system.
The invention is realized in this way a kind of smart city big data processing system based on Hadoop aggregated structure, institute
Stating the smart city big data processing system framework based on Hadoop aggregated structure includes:
Acquisition layer: connecting with inclusion layer, for acquiring data；
Inclusion layer: connecting with acquisition layer, application layer, for carrying out aggregation processing to the data of acquisition, and to a variety of data
The data in source carry out unified loading, classification, processing and storage；
Application layer: connecting with inclusion layer, is modeled using algorithms of different and is set using mobile device, plate, computer or terminal
It is standby that each functional module is presented.
Further, the acquisition layer specifically includes: data acquisition interface, Hadoop cluster, relevant database.Data are adopted
Collection interface: being to acquire data using transposition from exterior and be input to an interface of internal system.Hadoop cluster:
Hadoop is the distributed parallel programming framework that can run on large-scale cluster of open source, and most crucial design includes:
The design of MapReduce and HDFS.Relevant database: relational database is the Database Systems for supporting relational model, by closing
It is data structure, relational operation set and integrity constraint three parts composition.
Further, the data acquisition interface specifically includes: 2G interface, 3G interface, 4G interface；The 2G interface includes:
GB/GN/A interface；The 3G interface includes: IU/PS/GN/A interface；The 4G interface includes: S1/X2/S6a/S5 interface；
The Hadoop cluster includes Hbase/Hive/Hdfs core component.
Further, the inclusion layer specifically includes:
SQL interface, Spark service, ROLAP Server, SPARK memory accelerate computing engines, Spark stream,
Spark SQL, Gaplx/mllib model algorithm library.SQL Server network interface is built upon between client and server
Network connection protocol layer.Spark is general, expansible distributed computing engine.
ROLAP indicates that the OLAP based on relational database realizes (Relational OLAP).Using relational database as core
The heart, with relational structures carry out multidimensional data indicate and storage.The multidimensional structure of multi-dimensional database is divided into two classes by ROLAP
Table: one kind is true table, is used to storing data and dimension keyword；Another kind of is dimension table, i.e., at least uses a table to each dimension
Come the description information of the dimensions such as level, the member's classification of storing dimension.SPARK memory accelerates computing engines: similar with Hadoop to open
Source cluster computing environment, Spark enables memory distributed data collection, and other than being capable of providing interactive inquiry, it can be with excellent
Change iteration workload.
Spark Stream is that streaming computing is resolved into a series of short and small batch processing jobs, input data according to
Batch size (such as 1 second) is divided into sectional data (DStream), and every one piece of data is all converted into the RDD in Spark,
Then the Transformation of DStream operation will be become operating the Transformation of RDD in Spark,
RDD is become intermediate result by operation to save in memory.
Spark SQL Spark SQL uses the processing of SQL statement and relevant database to the processing of SQL statement
Similar method can be parsed SQL statement (Parse) first, then form a Tree.SQL statement passes through first
Parser module is resolved to syntax tree.
Gaplx/mllib model algorithm library is realization library of the Spark to common machine learning algorithm, while including correlation
Test and Data Generator.
Further, the SQL interface includes JDBC/ODBC interface.
The Spark service includes Scala/Java/Python.
Further, the application layer specifically includes terminal device and functional module.
The terminal device includes: mobile device, plate, computer or other terminal devices.
The functional module includes: system management module, report development platform, mobile BI, event development platform, multidimensional point
Analyse platform, self-service analysis platform, Platform for Visual Design of Vehicle, Visual Data Mining Platform in Electricity.
Each platform is in communication with each other by way of WebSocket with server, front end, and SDK can periodically receive service
The page request that device issues；Then page snapshot and interface factor information can be reported to server, server receives meeting after information
It is analyzed according to each element of the interface factor information to the page, being marked which page elements to be according to the type of control can
To be buried a little；Information can finally be buried and give front end rendering, at this point, showing in the Web page of front end just is exactly that can bury
The page of point.
Another object of the present invention is to provide a kind of smart city big data processing side based on Hadoop aggregated structure
Method, comprising:
Acquire the data of multiple data sources.
Aggregation processing carried out to the data of acquisition, and to the data of multiple data sources carry out unified loading, classification, processing with
And storage.
It is modeled using algorithms of different and is presented using mobile device, plate, computer or terminal device.
Further, aggregation processing is carried out to the data of acquisition, and unified loading is carried out to the data of multiple data sources, is divided
In class, processing and storage, specifically includes the following steps:
1) file prepares.
2: file slice, cutting is three pieces.
3) Map is run: being received a key-value pair, is generated one group of key-value pair.
4) distribute: key-value pair is distributed to Reduce by Shuffle.
5) Reduce is run: the value of same keys is added up.
6) calculated result is exported.
Further, in being presented after modeling and using mobile device, plate, computer or terminal device, each function mould
Block is in communication with each other by way of WebSocket with server, front end, and SDK timing, which receives the page that server issues, asks
It asks.
Then report page snapshot and interface factor information to server, server can be according to the interface factor after receiving information
Information analyzes each element of the page, and it is to be buried a little which page elements is marked according to the type of control.
Information can finally be buried and give front end rendering, show it is the page that can be buried a little in the Web page of front end.
Another object of the present invention is to provide the big numbers in smart city based on Hadoop aggregated structure described in a kind of carrying
According to the smart city big data processing platform based on Hadoop aggregated structure of processing system.
In conclusion advantages of the present invention and good effect are as follows:
The present invention solves in survey region that crowd's flowing is irregular, and regional analysis is difficult the technical problem carried out: city
Structure uses GIS spatial analysis mode, relates generally to the data converting function of generalized information system, the editting function of graph data passes through
The regularity of distribution of vector graphics or grating image space is analyzed, wherein vector quantization process is most important to regional analysis
One step.
The present invention achieves unexpected technical effect, specifically includes:
(1) present invention has superior technique effect compared with prior art, and each Regional Population Floating can be predicted in the present invention,
Reasonable distribution social resources and the strategy of sustainable development are inquired into, provides data supporting for government decision, platform architecture inclusion layer is adopted
Implemented with hadoop framework, development cost can be effectively reduced, there is preferable execution efficiency and scalability.
(2) invention, which represents, uses hadoop architecture technology development trend, by feat of the storage of super large file, unified file system
Unite access interface, file block storage, high fault tolerance enhancing system performance.
(3) domestic technique blank has been filled up, in the prior art, mobile subscriber's signaling data has been not based on and predicts each region
Movement of population, cannot effectively analyze social resources can distribute data, and without using hadoop framework, cannot be effectively reduced open
Send out cost；Not using SPARK open source parallel architecture, speed of service speed in data mining and machine learning is caused.The present invention
The analysis difficult problem of the user data of very good solution magnanimity, low-quality can be very good analysis mobility status
The present invention is based on the smart city big data processing systems of Hadoop aggregated structure, are based on mobile subscriber's signaling data
(the mobile phone user's accounting moved in three big operators mainly extracts the mobile phone user of commmunication company up to 80% or so), prediction is each
Regional Population Floating inquires into reasonable distribution social resources and the strategy of sustainable development, provides data supporting for government decision, have
Preferable execution efficiency and scalability.
Platform architecture inclusion layer of the present invention is implemented using hadoop framework, development cost can be effectively reduced, and form one
Body application and development has very strong scalability.
The present invention compares hadoop MapReduce, Spark can be fitted preferably using SPARK open source parallel architecture design
The algorithm of the MapReduce of iteration is needed for data mining and machine learning etc., and upper hand is easy, the speed of service is quick.
Detailed description of the invention
Fig. 1 is the smart city big data processing system framework provided in an embodiment of the present invention based on Hadoop aggregated structure
Schematic diagram.
In figure: 1, acquisition layer；2, inclusion layer；3, application layer；4, data acquisition interface；5, Hadoop cluster；6, relationship type
Database；7, SQL interface；8, Spark is serviced；9,ROLAP Server；10, SPARK memory accelerates computing engines；11,Spark
stream；12,Spark SQL；13, Gaplx/mllib model algorithm library；14, terminal device；15, functional module.
Fig. 2 is the smart city big data processing method process provided in an embodiment of the present invention based on Hadoop aggregated structure
Figure.
Fig. 3 is provided in an embodiment of the present invention to count in some file for tri- word quantity of Deer, Car and Bear
Illustrate MapReduce is how to realize that distributed storage calculates effect picture.
Specific embodiment
In order to make the objectives, technical solutions, and advantages of the present invention clearer, with reference to embodiments, to the present invention
It is further elaborated.It should be appreciated that the specific embodiments described herein are merely illustrative of the present invention, it is not used to
Limit the present invention.
Existing wisdom platform can not analyze movement of population；Magnanimity, low-quality user data to query analysis work bring
Lot of challenges；And in the prior art, it is not based on mobile subscriber's signaling data and predicts each Regional Population Floating, cannot effectively divide
Analysis social resources can distribute data, and without using hadoop framework, development cost cannot be effectively reduced；Do not use
SPARK open source parallel architecture, causes speed of service speed in data mining and machine learning.
To solve the above problems, being described in detail with reference to the accompanying drawing to application principle of the invention.
As shown in Figure 1, the smart city big data processing system provided in an embodiment of the present invention based on Hadoop aggregated structure
System specifically includes:
Acquisition layer 1, inclusion layer 2, application layer 3.
Acquisition layer 1: it is connect with inclusion layer 2, for acquiring data.
Inclusion layer 2: connecting with acquisition layer 1, application layer 3, for carrying out aggregation processing to the data of acquisition, realizes to a variety of
The data of data source carry out unified loading, classification, processing and storage.
Application layer 3: connecting with inclusion layer 2, for based on function difference using algorithms of different model and utilize mobile device,
Each functional module is presented in plate, computer or other terminal devices.
Acquisition layer 1 provided in an embodiment of the present invention specifically includes:
Acquisition layer 1 includes: data acquisition interface 4, Hadoop cluster 5, relevant database 6.
The data acquisition interface 4 specifically includes: 2G interface, 3G interface, 4G interface.The 2G interface includes: GB/GN/A
Interface.The 3G interface includes: IU/PS/GN/A interface.The 4G interface includes: S1/X2/S6a/S5 interface.
The Hadoop cluster 5 are as follows: using Hbase/Hive/Hdfs as the Hadoop cluster of core component.
Inclusion layer 2 provided in an embodiment of the present invention specifically includes:
Inclusion layer 2 includes: SQL interface 7, Spark service 8, ROLAP Server9, SPARK memory acceleration computing engines
10, Spark stream11, Spark SQL12, Gaplx/mllib model algorithm library 13.
The SQL interface 7 includes: JDBC/ODBC interface.
The Spark service 8 includes: Scala/Java/Python.
Application layer 3 provided in an embodiment of the present invention specifically includes:
Application layer specifically includes: terminal device 14 and functional module 15.
The terminal device 14 specifically includes: mobile device, plate, computer or other terminal devices.
The functional module 15 specifically includes: system management module, report development platform, mobile BI, event development platform,
Multidimensional analysis platform, self-service analysis platform, Platform for Visual Design of Vehicle, Visual Data Mining Platform in Electricity.
As shown in Fig. 2, the present invention provides a kind of smart city big data processing method based on Hadoop aggregated structure, packet
It includes:
Acquire the data of multiple data sources.
Aggregation processing carried out to the data of acquisition, and to the data of multiple data sources carry out unified loading, classification, processing with
And storage.
It is modeled using algorithms of different and is presented using mobile device, plate, computer or terminal device.Each function mould
Block is in communication with each other by way of WebSocket with server, front end, and SDK timing, which receives the page that server issues, asks
It asks.Then report page snapshot and interface factor information to server, server can be according to interface factor information after receiving information
Each element of the page is analyzed, it is to be buried a little which page elements is marked according to the type of control.Finally may be used
It buries information and gives front end rendering, show it is the page that can be buried a little in the Web page of front end.
In embodiments of the present invention, traditional date storage method is stored by file, number more than TB rank
According to being stored in a file by realizing and analyzing data, efficiency is very low, and here it is traditional numbers to file read-write operations
According to storage scheme.And the data volume that the mobile phone user in a present city generates should easily reach TB rank or more.
Hadoop aggregated structure is to solve the problems, such as this.Hadoop is stored using distributed document, to guarantee that file integrality is anti-
It only loses, it will usually back up 3 parts or more.Distributed document storage HDFS solves the storage problem and reading speed of big data
Problem.The distributed computing platform of Hadoop is MapReduce, for solving mass data computational problem.MapReduce has two
A stage forms Map and Reduce, and user only needs to realize the two functions, and distributed computing can be realized.To count some
In file for tri- word quantity of Deer, Car and Bear, as shown in Figure 3.Illustrate MapReduce is how to realize distribution
Storage calculates.
The invention will be further described combined with specific embodiments below.
Embodiment
Smart city big data processing method provided in an embodiment of the present invention based on Hadoop aggregated structure, comprising:
1, file prepares.
2, file is sliced: cutting here is three pieces (job parallelism processing, efficiency double).
3, Map process: receiving a key-value pair, generates one group of key-value pair, such as (Deer, 1), representing Deer as key, 1 is
Value, represents the quantity of Deer word.
4, distribute process: key-value pair is distributed to Reduce by Shuffle.
5, the value of same keys Reduce process: is added up into (calculating close data).
6, calculated result is exported.
Hadoop system is stored by HDFS and MapReduce, HDFS by feat of super large file, unified file system accesses
Interface, file block storage, high fault tolerance characteristic, MapReduce by feat of automatically parallelizing, automatic reliable treatments, flexibly
The characteristics such as extension, high-performance, the speed that the use of this two big core substantially increases data storage and reads, it is unnecessary to reduce
Resource consumption.
The foregoing is merely illustrative of the preferred embodiments of the present invention, is not intended to limit the invention, all in essence of the invention
Made any modifications, equivalent replacements, and improvements etc., should all be included in the protection scope of the present invention within mind and principle.
Claims (8)
1. a kind of smart city big data processing system based on Hadoop aggregated structure, which is characterized in that described to be based on
The smart city big data processing system of Hadoop aggregated structure includes:
Acquisition layer: connecting with inclusion layer, for acquiring data；
Inclusion layer: connecting with acquisition layer, application layer, for carrying out aggregation processing to the data of acquisition, and to multiple data sources
Data carry out unified loading, classification, processing and storage；
Application layer: connecting with inclusion layer, and each function mould is presented using mobile device, plate, computer or terminal device after modeling
Block.
2. the smart city big data processing system based on Hadoop aggregated structure as described in claim 1, which is characterized in that institute
It states acquisition layer to specifically include: data acquisition interface, Hadoop cluster, relevant database；
The data acquisition interface specifically includes: 2G interface, 3G interface, 4G interface；The 2G interface includes: GB/GN/A interface；
The 3G interface includes: IU/PS/GN/A interface；The 4G interface includes: S1/X2/S6a/S5 interface；
The Hadoop cluster includes Hbase/Hive/Hdfs core component.
3. the smart city big data processing system based on Hadoop aggregated structure as described in claim 1, which is characterized in that institute
Inclusion layer is stated to specifically include:
SQL interface, Spark service, ROLAP Server, SPARK memory accelerate computing engines, Spark stream, Spark
SQL, Gaplx/mllib model algorithm library；
The SQL interface includes JDBC/ODBC interface；For the network connection between client and server；
The Spark service includes Scala/Java/Python；For expansible distributed computing engine；
The ROLAP Server is used to indicate that the OLAP based on relational database to be realized；
The SPARK memory accelerates computing engines, open source cluster computing environment similar with Hadoop；
Streaming computing is resolved into a series of short and small batch processing jobs by the Spark Stream, input data according to
Batch size is divided into sectional data, and every one piece of data is all converted into the RDD in Spark, then will be to DStream's
Transformation operation becomes operating the Transformation of RDD in Spark, and RDD is become by operation
Intermediate result saves in memory；
The Spark SQL, the processing of processing and relevant database to SQL statement to SQL statement, SQL statement is carried out
Then parsing forms a Tree；
Gaplx/mllib model algorithm library, including relevant test and Data Generator.
4. the smart city big data processing system based on Hadoop aggregated structure as described in claim 1, which is characterized in that institute
It states application layer and specifically includes terminal device and functional module；
The terminal device includes: mobile device, plate, computer or other terminal devices；
The functional module includes: system management module, report development platform, mobile BI, event development platform, multidimensional analysis are flat
Platform, self-service analysis platform, Platform for Visual Design of Vehicle, Visual Data Mining Platform in Electricity.
5. a kind of smart city big data processing system based on Hadoop aggregated structure as described in claim 1 based on
The smart city big data processing method of Hadoop aggregated structure, which is characterized in that the intelligence based on Hadoop aggregated structure
Intelligent city big data processing method includes:
Acquire the data of multiple data sources；
Aggregation processing is carried out to the data of acquisition, and unified loading, classification are carried out to the data of multiple data sources, handles and deposits
Storage；
It carries out after modeling and using mobile device, plate, computer or terminal device that each functional module is presented.
6. the smart city big data processing method based on Hadoop aggregated structure as claimed in claim 5, which is characterized in that
Aggregation processing is carried out to the data of acquisition, and unified loading, classification, processing and storage are carried out to the data of multiple data sources
In, specifically includes the following steps:
1) file prepares；
2: file slice, cutting is three pieces；
3) Map is run: being received a key-value pair, is generated one group of key-value pair；
4) distribute: key-value pair is distributed to Reduce by Shuffle；
5) Reduce is run: the value of same keys is added up；
6) calculated result is exported.
7. the smart city big data processing method based on Hadoop aggregated structure as claimed in claim 5, which is characterized in that
In being presented after modeling and using mobile device, plate, computer or terminal device, each functional module passes through WebSocket
Mode and server, front end be in communication with each other, SDK timing receives the page request that server issues；
Then report page snapshot and interface factor information to server, server can be according to interface factor information after receiving information
Each element of the page is analyzed, it is to be buried a little which page elements is marked according to the type of control；
Information can finally be buried and give front end rendering, show it is the page that can be buried a little in the Web page of front end.
8. it is a kind of carry claim 1 described in the smart city big data processing system based on Hadoop aggregated structure based on
The smart city big data processing platform of Hadoop aggregated structure.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910827032.1A CN110532283A (en) | 2019-09-03 | 2019-09-03 | A kind of smart city big data processing system based on Hadoop aggregated structure |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910827032.1A CN110532283A (en) | 2019-09-03 | 2019-09-03 | A kind of smart city big data processing system based on Hadoop aggregated structure |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110532283A true CN110532283A (en) | 2019-12-03 |
Family
ID=68666335
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910827032.1A Pending CN110532283A (en) | 2019-09-03 | 2019-09-03 | A kind of smart city big data processing system based on Hadoop aggregated structure |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110532283A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111092938A (en) * | 2019-12-04 | 2020-05-01 | 重庆特斯联智慧科技股份有限公司 | Smart city management system based on cloud platform |
| CN111475545A (en) * | 2020-04-02 | 2020-07-31 | 湖南赛吉智慧城市建设管理有限公司 | City base database system for smart city |
| CN112035431A (en) * | 2020-07-17 | 2020-12-04 | 中国城市规划设计研究院 | Construction processing method and system for universal data format of smart city |
| CN113727330A (en) * | 2020-05-21 | 2021-11-30 | 大唐移动通信设备有限公司 | User information processing method and device based on TD-LTE mobile communication system |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105677752A (en) * | 2015-12-30 | 2016-06-15 | 深圳先进技术研究院 | Streaming computing and batch computing combined processing system and method |
| US9646088B1 (en) * | 2011-06-30 | 2017-05-09 | Sumo Logic | Data collection and transmission |
| CN107818163A (en) * | 2017-11-01 | 2018-03-20 | 平安科技（深圳）有限公司 | Page display method, device, computer equipment and storage medium |
| CN107945086A (en) * | 2017-11-17 | 2018-04-20 | 广州葵翼信息科技有限公司 | A kind of big data resource management system applied to smart city |
| CN108022034A (en) * | 2016-11-04 | 2018-05-11 | 浙江奇道网络科技有限公司 | A kind of comprehensive executive decision is supported and administrative behavior management information platform |
| CN108985531A (en) * | 2017-06-01 | 2018-12-11 | 中国科学院深圳先进技术研究院 | A kind of multimode isomery electric power big data convergence analysis management system and method |
-
2019
- 2019-09-03 CN CN201910827032.1A patent/CN110532283A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9646088B1 (en) * | 2011-06-30 | 2017-05-09 | Sumo Logic | Data collection and transmission |
| CN105677752A (en) * | 2015-12-30 | 2016-06-15 | 深圳先进技术研究院 | Streaming computing and batch computing combined processing system and method |
| CN108022034A (en) * | 2016-11-04 | 2018-05-11 | 浙江奇道网络科技有限公司 | A kind of comprehensive executive decision is supported and administrative behavior management information platform |
| CN108985531A (en) * | 2017-06-01 | 2018-12-11 | 中国科学院深圳先进技术研究院 | A kind of multimode isomery electric power big data convergence analysis management system and method |
| CN107818163A (en) * | 2017-11-01 | 2018-03-20 | 平安科技（深圳）有限公司 | Page display method, device, computer equipment and storage medium |
| CN107945086A (en) * | 2017-11-17 | 2018-04-20 | 广州葵翼信息科技有限公司 | A kind of big data resource management system applied to smart city |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111092938A (en) * | 2019-12-04 | 2020-05-01 | 重庆特斯联智慧科技股份有限公司 | Smart city management system based on cloud platform |
| CN111475545A (en) * | 2020-04-02 | 2020-07-31 | 湖南赛吉智慧城市建设管理有限公司 | City base database system for smart city |
| CN113727330A (en) * | 2020-05-21 | 2021-11-30 | 大唐移动通信设备有限公司 | User information processing method and device based on TD-LTE mobile communication system |
| CN112035431A (en) * | 2020-07-17 | 2020-12-04 | 中国城市规划设计研究院 | Construction processing method and system for universal data format of smart city |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110532283A (en) | A kind of smart city big data processing system based on Hadoop aggregated structure | |
| CN105139281A (en) | Method and system for processing big data of electric power marketing | |
| Ben Brahim et al. | Spatial data extension for Cassandra NoSQL database | |
| CN109582667A (en) | A kind of multiple database mixing storage method and system based on power regulation big data | |
| CN105631764A (en) | Smart power grid big data application system orienting smart city | |
| Xia et al. | Big traffic data processing framework for intelligent monitoring and recording systems | |
| CN109582717A (en) | A kind of database unified platform and its read method towards electric power big data | |
| Bakli et al. | HadoopTrajectory: a Hadoop spatiotemporal data processing extension | |
| Vaisman et al. | Data warehouses: Next challenges | |
| Ding et al. | Massive heterogeneous sensor data management in the Internet of Things | |
| CN111753034A (en) | One-stop type geographical big data platform | |
| CN104778355B (en) | The abnormal track-detecting method of traffic system is distributed based on wide area | |
| CN109753541A (en) | A kind of relational network construction method and device, computer readable storage medium | |
| CN113254517A (en) | Service providing method based on internet big data | |
| Ali et al. | From big data to smart data-centric software architectures for city analytics: the case of the pell smart city platform | |
| Budde et al. | Leveraging spatio-temporal clustering for participatory urban infrastructure monitoring | |
| Lee et al. | Scalable activity-travel pattern monitoring framework for large-scale city environment | |
| CN110019466A (en) | Big data integrated system based on metadata | |
| Wang et al. | GIS cloud computing based government Big Data analysis platform | |
| Liu et al. | Research on the clustering algorithm of the bicycle stations based on OPTICS | |
| CN111291102A (en) | High-performance scale statistical calculation method for government affair data mining | |
| Zaragozí et al. | Towards an Affordable GIS for Analysing Public Transport Mobility Data: A Preliminary File Naming Convention for Avoiding Duplication of Efforts. | |
| Menezes et al. | Challenges in RFID deployment–a case study in public transportation | |
| CN110516021A (en) | A kind of mobile phone user flowing law analysis method and system based on big data | |
| Li et al. | Analysis and research on the use situation of public bicycles based on spark machine learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20191203 |
|
| RJ01 | Rejection of invention patent application after publication |