CN107133273A - A kind of transit's routes data processing method and server cluster based on big data - Google Patents
A kind of transit's routes data processing method and server cluster based on big data Download PDFInfo
- Publication number
- CN107133273A CN107133273A CN201710225226.5A CN201710225226A CN107133273A CN 107133273 A CN107133273 A CN 107133273A CN 201710225226 A CN201710225226 A CN 201710225226A CN 107133273 A CN107133273 A CN 107133273A
- Authority
- CN
- China
- Prior art keywords
- data
- server
- business datum
- memory
- server cluster
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 12
- 238000012545 processing Methods 0.000 claims abstract description 71
- 238000013480 data collection Methods 0.000 claims abstract description 27
- 238000000034 method Methods 0.000 claims abstract description 16
- 238000003860 storage Methods 0.000 claims abstract description 15
- 238000004458 analytical method Methods 0.000 claims description 30
- 238000012517 data analytics Methods 0.000 claims description 17
- 238000006243 chemical reaction Methods 0.000 claims description 15
- 238000007405 data analysis Methods 0.000 claims description 10
- 230000000007 visual effect Effects 0.000 claims description 8
- 238000004140 cleaning Methods 0.000 claims description 7
- 238000009826 distribution Methods 0.000 claims description 2
- 239000004744 fabric Substances 0.000 claims 2
- 230000008569 process Effects 0.000 abstract description 9
- 238000012423 maintenance Methods 0.000 abstract description 4
- 230000001360 synchronised effect Effects 0.000 abstract description 4
- 238000007726 management method Methods 0.000 description 10
- 230000006870 function Effects 0.000 description 8
- 238000010586 diagram Methods 0.000 description 6
- 238000004590 computer program Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 5
- 238000012544 monitoring process Methods 0.000 description 5
- 238000011161 development Methods 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 238000010276 construction Methods 0.000 description 3
- 238000013500 data storage Methods 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000013523 data management Methods 0.000 description 2
- 238000012216 screening Methods 0.000 description 2
- 238000007619 statistical method Methods 0.000 description 2
- 241001269238 Data Species 0.000 description 1
- 241001310793 Podium Species 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000012098 association analyses Methods 0.000 description 1
- 238000009412 basement excavation Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000007418 data mining Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 238000005194 fractionation Methods 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 238000003032 molecular docking Methods 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 238000004321 preservation Methods 0.000 description 1
- 230000011664 signaling Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
- 230000003442 weekly effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/283—Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G06Q50/40—
Abstract
The embodiment of the present invention provides a kind of transit's routes data processing method and server cluster based on big data, server cluster obtains the business datum of each track circuit in Metro Network from data collection layer, then the business datum to acquisition is handled and stored, and the traffic circulation state of each track circuit in Metro Network is controlled finally according to the business datum after processing.The embodiment of the present invention disposes big data platform by building server cluster, substitute data processing and storage that existing Data Warehouse Platform carries out gauze (emergent) command centre, built due to server cluster and safeguard more general convenient, be easy to system to dispose first and later maintenance.The memory capacity of other server cluster can realize that linear transverse extends, process performance also can Synchronous lifting, need to only increase machine into cluster, and expanding course effectively meets the demand of dilatation convenience without shutting down.
Description
Technical field
The present embodiments relate to track transit's routes Center Technology field, more particularly to it is a kind of based on big data
Transit's routes data processing method and server cluster.
Background technology
With the continuous construction and development of domestic and international urban track traffic, city rail traffic route plans increasing fast,
Built and the demand that operation efficiency is lifted constantly is strengthened building circuit, the construction demand of gauze (emergent) command centre increasingly increases
By force, the data volume of gauze (emergent) command centre is also continuously increased.Existing gauze (emergent) command centre generally employs data
Stores Stressed Platform adds supporting management software to realize the function of data storage and processing, but the general frame of available data Stores Stressed Platform
It is located on special equipment, environmental structure is complicated, and downtime is longer during expansion.Data warehouse is when doing linear expansion, and data are needed
Redistribution is wanted, consumption resource is big, and the time is long, it is impossible to low cost reply gauze (emergent) command centre's built by separate periods or later stage dilatation
Demand.
The content of the invention
The embodiment of the present invention provides a kind of transit's routes data processing method and server cluster based on big data, is used for
Solve Data Warehouse Platform in the prior art and build the problem of complicated and later stage autgmentability is weak.
The embodiments of the invention provide a kind of transit's routes data processing method based on big data, including：
Server cluster obtains the business datum of each track circuit in Metro Network from data collection layer；
The server cluster is handled and stored to the business datum of acquisition；
Friendship of the server cluster according to the business datum after processing to each track circuit in the Metro Network
Logical running status is controlled.
Alternatively, the server cluster includes data acquisition server and interface server；
The server cluster obtains the business datum of each track circuit in Metro Network from data collection layer, bag
Include：The server cluster obtains the business datum of each track circuit by the interface server from data collection layer,
The business datum of each track circuit be by the data acquisition server from the service sub-system of each track circuit
Gather and be stored in data collection layer.
Alternatively, the server cluster includes data processing server and memory；
The server cluster is handled and stored to the business datum of acquisition, including：
The data processing server carries out data pick-up and data cleansing to the business datum of collection；
Business datum after cleaning is carried out data conversion by the data processing server；
The data processing server preserves the business datum after data conversion into the memory.
Alternatively, the memory includes real-time memory and distributed memory；
The data processing server preserves the business datum after data conversion into the memory, including：
When the business datum is the data for needing real-time release, the data for needing real-time release are stored in described
In real-time memory；
After it is determined that the data for needing real-time release are issued, the data for needing real-time release are preserved
Into the distributed memory；
The business datum preserves the data for not needing real-time release to institute not need during real-time release data
State in distributed memory.
Alternatively, the server cluster includes data analytics server and control centre's server；
Friendship of the server cluster according to the business datum after processing to each track circuit in the Metro Network
Logical running status is controlled, including：
The data analytics server sets up analysis model by visual logical edit instrument；
The data analytics server is carried out according to the analysis model to the business datum in the distributed memory
Data analysis；
Friendship of the control centre's server based on data analysis result to each track circuit in the Metro Network
Logical running status is controlled.
Correspondingly, the embodiment of the present invention additionally provides a kind of server cluster, including：
Acquisition module, the business datum for obtaining each track circuit in Metro Network from data collection layer；
Processing module, is handled and is stored for the business datum to acquisition；And according to the business datum after processing
Traffic circulation state to each track circuit in the Metro Network is controlled.
Alternatively, the acquisition module includes data acquisition server and interface server；
The acquisition module specifically for：
The business datum of each track circuit, each described rail are obtained from data collection layer by the interface server
The business datum of road circuit is gathered and preserved from the service sub-system of each track circuit by the data acquisition server
In data collection layer.
Alternatively, the processing module includes data processing server and memory；
The processing module specifically for：
Data pick-up and data cleansing are carried out to the business datum of collection by the data processing server；
Business datum after cleaning is carried out by data conversion by the data processing server；
The business datum after data conversion is preserved into the memory by the data processing server.
Alternatively, the memory includes real-time memory and distributed memory；
The processing module specifically for：
When the business datum is the data for needing real-time release, the data for needing real-time release are stored in described
In real-time memory；
After it is determined that the data for needing real-time release are issued, the data for needing real-time release are preserved
Into the distributed memory；
The business datum preserves the data for not needing real-time release to institute not need during real-time release data
State in distributed memory.
Alternatively, the processing module includes data analytics server and control centre's server；
The processing module specifically for：
The data analytics server sets up analysis model by visual logical edit instrument；
The data analytics server is carried out according to the analysis model to the business datum in the distributed memory
Data analysis；
Friendship of the control centre's server based on data analysis result to each track circuit in the Metro Network
Logical running status is controlled.
The embodiment of the present invention shows that server cluster obtains each track circuit in Metro Network from data collection layer
Business datum, then the business datum to acquisition handled and stored, finally according to the business datum after processing to track
The traffic circulation state of each track circuit is controlled in transit's routes.The embodiment of the present invention is by building server cluster come portion
Big data platform is affixed one's name to, data processing and storage that existing Data Warehouse Platform carries out gauze (emergent) command centre is substituted, by
Built in server cluster and safeguard more general convenient, be easy to system to dispose first and later maintenance.Other server cluster
Memory capacity can realize that linear transverse extends, process performance also can Synchronous lifting, only need to increase machine into cluster, and expand
Process effectively meets the demand of dilatation convenience without shutting down.
Brief description of the drawings
Technical scheme in order to illustrate the embodiments of the present invention more clearly, makes required in being described below to embodiment
Accompanying drawing is briefly introduced, it should be apparent that, drawings in the following description are only some embodiments of the present invention, for this
For the those of ordinary skill in field, without having to pay creative labor, it can also be obtained according to these accompanying drawings
His accompanying drawing.
Fig. 1 is a kind of big data platform system structure figure provided in an embodiment of the present invention；
Fig. 2 illustrates for a kind of flow of transit's routes data processing method based on big data provided in an embodiment of the present invention
Figure；
Fig. 3 is a kind of schematic flow sheet of data processing method provided in an embodiment of the present invention；
Fig. 4 is a kind of gauze command center system Organization Chart based on big data platform provided in an embodiment of the present invention；
Fig. 5 is a kind of structural representation of server cluster provided in an embodiment of the present invention.
Embodiment
In order that the purpose of the present invention, technical scheme and beneficial effect are more clearly understood, below in conjunction with accompanying drawing and implementation
Example, the present invention will be described in further detail.It should be appreciated that specific embodiment described herein is only to explain this hair
It is bright, it is not intended to limit the present invention.
Technical scheme in the embodiment of the present invention can be applied to gauze command centre, pass through the transit's routes based on big data
Data processing method builds the system architecture of gauze command centre, mainly includes：Source data layer, data collection layer, data platform
Layer, service aid layer, application layer, can be realized in terms of hardware realization by building server cluster, specifically, source data layer
In be gauze command centre rely on data source, mainly including service sub-systems such as comprehensive monitoring system, allocation settlement centers.Number
It is responsible for according to acquisition layer from source data layer gathered data, mainly by data acquisition server, interface server, interface software and networking
Gateway etc. is constituted.Data platform layer is responsible for from interface server gathered data, carries out data processing, data preparation and history and deposits
Storage, including real-time data base and big data platform, wherein big data platform are included with lower module：System deployment and management, data
Storage, resource management handles engine, safety, data management, tool storage room and access interface.User can be in one-stop big number
According to collection, storage, analysis, search, excavation mass data and its inherent value on comprehensive platform.Below with specific embodiment introduction
The architectural framework of big data platform, as shown in figure 1, the architectural framework of big data platform includes the one-stop He of big data platform 101
102 two parts of big data infrastructure, wherein one-stop big data platform specifically include Flume (result collection system),
Sqoop2, data management, system administration, batch processing, interactive analysis, search engine, machine learning, stream process, third party answer
With, unified resource management, distributed file system, distributed memory system and structuring, semi-structured, unstructured data
Storage；Big data infrastructure specifically includes server, safety means and network.Service aid layer is responsible for providing background application clothes
Business, corresponding service is set according to different business demand, and provides various application tools, is mainly made up of application server.Should
It is responsible for realizing different gauze business with layer, including runs the aobvious of commander, emergency command, information service, operation assessment and large-size screen monitors
Show etc..
Fig. 2 illustrates a kind of transit's routes data processing side based on big data provided in an embodiment of the present invention
Method, the flow can be performed by server cluster.
As shown in Fig. 2 the specific steps of the flow include：
Step S201, server cluster obtains the business number of each track circuit in Metro Network from data collection layer
According to.
Step S202, server cluster is handled and stored to the business datum of acquisition.
Step S203, friendship of the server cluster according to the business datum after processing to each track circuit in Metro Network
Logical running status is controlled.
Specifically, in step s 201, server cluster obtains each rail by interface server from data collection layer
The business datum of road circuit, the business datum of each track circuit is the business by data acquisition server from each track circuit
Gather and be stored in data collection layer in subsystem, wherein service sub-system is included in comprehensive monitoring system, allocation settlement
The heart, passenger information system, close-circuit television,closed-circuft televishon etc., corresponding business datum include passenger flow data, device data, power supply number
According to, travelling data etc..
In step S202, server cluster includes data processing server and memory, passes through data processing server
Business datum to acquisition is handled, as shown in figure 3, being specially：Data processing server is carried out to the business datum of collection
Data pick-up and data cleansing, then carry out data conversion and loading by the business datum after cleaning, and the data to loading are carried out
Preserved after Data Quality Analysis into memory.Above-mentioned data handling procedure supports the number of the heterogeneous data source of various platforms
According to, including architectural system and non-structured data；The different editions of various relevant databases are also supported, for not
The perform script accordingly optimized is generated with version.In specific implementation, using ETL (Extract-Transform-Load, abbreviation number
According to REPOSITORY TECHNOLOGY) instrument realizes unified data dispatch and data integrated management, and pass through the development scheme of towed, reduce number
According to integrated exploitation complexity, following functions are specifically included：
1st, built-in various integrated adapters, support is connected (Java Data Base with message queue, java databases
Connectivity, abbreviation jdbc), HTTP (HyperText Transfer Protocol, abbreviation HTTP),
The docking of all kinds of dominant systems frameworks such as distributed file system (Hadoop Distributed File System, HDFS),
Instrument also allows to carry out the function of adapter self-defined exploitation extension simultaneously.
2nd, the various data converting functions such as association, screening, field mapping, the field fractionation of data are supported, data are met clear
Wash, data conversion, the demand of data quality management.
3rd, built-in ripe scheduling engine and journal engine, support to be scheduled operation program management and monitoring alarm.
4th, the development interface of towed.From all kinds of peripheral systems are connected, to the mode for setting data processing, all by visual
The component of change is pulled and the mode of configuration is completed, and development and implementation speed is fast.
Further, data processing server preserves data after treatment in memory, wherein memory bag
Real-time memory and distributed memory are included, it is also different according to the position of the different storages of data character, be specially：
When business datum is the data for needing real-time release, it would be desirable to which the data of real-time release are stored in real-time memory
In.After it is determined that needing the data of real-time release issued, it would be desirable to which the data of real-time release are preserved to distributed storage
In device.Business datum preserves the data for not needing real-time release to distributed memory not need during real-time release data
In.In addition for the larger real time data of data volume, in order to ensure query performance, data deposit real-time data base, but only protect
A nearly weekly data is stayed, for platform data quality analysis, distributed data base is subsequently restored again into.By the analysis of conversion, cleaning
Data, are stored in distributed data base, are analyzed during with support sizes factually.Video data does not have excessive demand to read or write speed, uses
IP SAN (IP Storage Area Network, storage area network) are stored in distributed memory.
In step S203, server cluster includes data analytics server and control centre's server.Server cluster
The traffic circulation state of each track circuit in Metro Network is controlled according to the business datum after processing, is specially：
Data analytics server sets up analysis model by visual logical edit instrument, and then distribution is deposited according to analysis model
Business datum in reservoir carries out data analysis.Last control centre server based on data analysis result is to Metro Network
The traffic circulation state of interior each track circuit is controlled.In specific implementation, analysis model is to realize number based on big data platform
According to freely the screening of table, multidimensional combination, collision, pass through what visual logical edit instrument was set up.Logic knot between tables of data
Structure at least include association, it is left association, it is right association, it is right exclude, left bank is removed, duplicate removal merges, whole merges, repel merge 8 kinds of data
Collision mode, while built-in Ensemble classifier and recurrence, time series pattern, clustering, association analysis, the magnitude of traffic flow (Origin-
Destination, abbreviation OD) analysis scheduling algorithm.Complete analysis model set up after, can a key initiate data analysis task, only need
There is provided data source, data parameters just can complete analysis, and system supports EXCEL data to import simultaneously in terms of data source.Complete to divide
After analysis, system background is scheduled management to task, and returns result to user.Operating personnel can be to querying condition, data
Table carries out drag operation.Complete after interface editing, data result can be led as word or EXCEL forms, while can basis
Need the module being published to and apply fairground.Managed using fairground using special topicization, including statistical analysis fairground etc., in application collection
Can be transferred in city as the function menu of normalization by specified analysing content to be published in the form of the page in platform,
Using.Control centre's server issues data or progress operation commander, emergency command by external data interface to outside system
Deng.External data interface can be divided into two kinds of database interface and file interface, and database interface is present in external interface data
In the interface table in storehouse；External interface file is present in the predetermined region of file server.The embodiment of the present invention is by building clothes
It is engaged in device cluster to dispose big data platform, substitutes existing Data Warehouse Platform and carry out at the data of gauze (emergent) command centre
Reason and storage, are built due to server cluster and safeguard more general convenient, be easy to system to dispose first and later maintenance.Take in addition
Business device cluster memory capacity can realize linear transverse extend, process performance also can Synchronous lifting, only need to increase machine into cluster
Device, and expanding course effectively meets the demand of dilatation convenience without shutting down.
The configuration of big data platform cluster is introduced with specific embodiment below, annual total initial data is set as 100TB,
Data deposit 3 copies, and disk is effectively calculated using space 70% and (avoided high water level), big data platform cluster configuration such as table 1
It is shown：
The big data platform cluster allocation list of table 1
Wherein, 24 pieces of 2TB serial port hard disks of single node (Serial Advanced Technology Attachment, abbreviation
SATA data disks) are done, it is considered to concurrent reading and writing and reliability requirement, its available capacity is 24*2/1.093/1.1*0.70=
27.95TB, the then nodes needed：100*3/27.95=11 platform.3 cluster management control nodes are additionally needed, amounting to needs
Nodes：14, obtain cluster management control node and storage calculate node hardware configuration is as shown in table 2：
The cluster management control node of table 2 and storage calculate node hardware configuration list
According to above-mentioned cluster configuration, compared to available data Stores Stressed Platform scheme, cost in the case of same disposal ability
Reduce 8-10 times, additionally by by ripe big data platform construction on general commercial server, it is possible to achieve hardware without
Guan Xing, that is, support physical machine arrangement, virtual machine arrangement, supports independent arrangement, cloud platform arrangement, can be closed according to requirements of the owner
Reason builds server cluster, at utmost reduces cost.
In order to preferably explain the embodiment of the present invention, describe the embodiment of the present invention below by specific implement scene and provide
A kind of transit's routes data processing method based on big data flow, set the gauze command centre based on big data platform
System architecture as shown in figure 4,
Gauze command center system framework based on big data platform includes source data layer 401, data collection layer 402, number
According to podium level 403, service aid layer 404 and application layer 405.Source data layer 401 includes each service sub-system, such as sorting knot
Calculation center (AFC Clearing Center, abbreviation ACC), comprehensive monitoring system (Integrated Supervisory
Control System, abbreviation ISCS), signal system (SIGnalling, abbreviation SIG), main transformer station, closed-circuit television monitoring system
Unite (Closed Circuit Television, CCTV), passenger information system (Passenger Information System,
Abbreviation PIS) and other service sub-systems.Data collection layer 402 is responsible for, from source data 401 capturing service data of layer, specifically including
Passenger flow data is gathered from ACC, from ISCS collecting device data, travelling data is gathered from SIG, from main transformer station's collection power supply number
According to from CCTV collection video datas, from PIS collection PCC (road network Bian Bo centers) data.After the gathered data of data collection layer 402,
By the real-time data base for needing the real-time data in large screen display to be saved in data platform layer 403, then using service aid
Instrument in layer 404 carries out the display on giant-screen after handling in real time, such as travelling data and passenger flow data need real-time display,
Then first be stored in real-time data base, then using service aid layer driving supervision and passenger flow supervision instrument handled after
Shown on giant-screen.In addition when device data and power data occur abnormal, device data and power data are preserved to reality
When database, then using the equipment supervision in service aid layer 404, power supply supervision and emergency processing instrument by device data
After being analyzed with power data on giant-screen Realtime Alerts.Business datum in real-time data base preserves a period of time standby
Part to big data platform, the business datum of real-time release is not needed directly to preserve to big data platform in data collection layer 402.Enter
One step, big data platform is stored to data, analyzed, being searched for, excavating mass data and its inherent value etc..Specific implementation
In, big data platform is analyzed the business datum of preservation using the analysis tool in service aid layer 404, analysis tool bag
Include GIS-Geographic Information System (Geographic Information System, abbreviation GIS) instrument, emulation tool, multidimensional analysis,
Business intelligence (Business Intelligence, abbreviation BI) displaying, data mining, passenger flow estimation, assessment algorithm.Analysis
Process is shown including analyze data modelling, task scheduling, model issue, using the major part of fairground management 4.The result of analysis
It can be assessed and information service specifically for operation commander, emergency command, statistical analysis, operation.In the embodiment of the present invention, based on big
Data platform is designed, and can effectively improve the processing capability in real time of data, meet the on-line storage of Various types of data, quick-searching,
The comprehensive demand such as real-time calculating and mining analysis.For the burst thing such as sudden mass incident, natural calamity and attack of terrorism
The emergency disposal ability of part is strong, effectively meets business demand.
Based on same idea, the structure for showing a kind of server cluster provided in an embodiment of the present invention exemplary Fig. 5,
The server cluster can perform the flow of the transit's routes data processing method based on big data.
Acquisition module 501, the business datum for obtaining each track circuit in Metro Network from data collection layer；
Processing module 502, is handled and is stored for the business datum to acquisition；And according to the business number after processing
It is controlled according to the traffic circulation state to each track circuit in the Metro Network.
Alternatively, the acquisition module 501 includes data acquisition server 5011 and interface server 5012；
The acquisition module 501 specifically for：
The business datum of each track circuit is obtained from data collection layer by the interface server 5012, it is described each
The business datum of individual track circuit is to be adopted by the data acquisition server 5011 from the service sub-system of each track circuit
Collect and be stored in data collection layer.
Alternatively, the processing module 502 includes data processing server 5021 and memory 5022；
The processing module 502 specifically for：
Data pick-up and data cleansing are carried out by the business datum of 5021 pairs of collections of data processing server；
Business datum after cleaning is carried out by data conversion by the data processing server 5021；
The business datum after data conversion is preserved to the memory by the data processing server 5021
In 5022.
Alternatively, the memory 5022 includes real-time memory and distributed memory；
The processing module 502 specifically for：
When the business datum is the data for needing real-time release, the data for needing real-time release are stored in described
In real-time memory；
After it is determined that the data for needing real-time release are issued, the data for needing real-time release are preserved
Into the distributed memory；
The business datum preserves the data for not needing real-time release to institute not need during real-time release data
State in distributed memory.
Alternatively, the processing module 502 includes data analytics server 5023 and control centre's server 5024；
The processing module 502 specifically for：
The data analytics server 5023 sets up analysis model by visual logical edit instrument；
The data analytics server 5023 is according to the analysis model to the business datum in the distributed memory
Carry out data analysis；
Control centre's server 5024 is according to data results to each track circuit in the Metro Network
Traffic circulation state be controlled.
The embodiment of the present invention shows that server cluster obtains each track circuit in Metro Network from data collection layer
Business datum, then the business datum to acquisition handled and stored, finally according to the business datum after processing to track
The traffic circulation state of each track circuit is controlled in transit's routes.The embodiment of the present invention is by building server cluster come portion
Big data platform is affixed one's name to, data processing and storage that existing Data Warehouse Platform carries out gauze (emergent) command centre is substituted, by
Built in server cluster and safeguard more general convenient, be easy to system to dispose first and later maintenance.Other server cluster
Memory capacity can realize that linear transverse extends, process performance also can Synchronous lifting, only need to increase machine into cluster, and expand
Process effectively meets the demand of dilatation convenience without shutting down.
The present invention is the flow with reference to method according to embodiments of the present invention, equipment (system) and computer program product
Figure and/or block diagram are described.It should be understood that can be by every first-class in computer program instructions implementation process figure and/or block diagram
Journey and/or the flow in square frame and flow chart and/or block diagram and/or the combination of square frame.These computer programs can be provided
The processor of all-purpose computer, special-purpose computer, Embedded Processor or other programmable data processing devices is instructed to produce
A raw machine so that produced by the instruction of computer or the computing device of other programmable data processing devices for real
The device for the function of being specified in present one flow of flow chart or one square frame of multiple flows and/or block diagram or multiple square frames.
These computer program instructions, which may be alternatively stored in, can guide computer or other programmable data processing devices with spy
Determine in the computer-readable memory that mode works so that the instruction being stored in the computer-readable memory, which is produced, to be included referring to
Make the manufacture of device, the command device realize in one flow of flow chart or multiple flows and/or one square frame of block diagram or
The function of being specified in multiple square frames.
These computer program instructions can be also loaded into computer or other programmable data processing devices so that in meter
Series of operation steps is performed on calculation machine or other programmable devices to produce computer implemented processing, thus in computer or
The instruction performed on other programmable devices is provided for realizing in one flow of flow chart or multiple flows and/or block diagram one
The step of function of being specified in individual square frame or multiple square frames.
, but those skilled in the art once know basic creation although preferred embodiments of the present invention have been described
Property concept, then can make other change and modification to these embodiments.So, appended claims are intended to be construed to include excellent
Select embodiment and fall into having altered and changing for the scope of the invention.
Obviously, those skilled in the art can carry out the essence of various changes and modification without departing from the present invention to the present invention
God and scope.So, if these modifications and variations of the present invention belong to the scope of the claims in the present invention and its equivalent technologies
Within, then the present invention is also intended to comprising including these changes and modification.
Claims (10)
1. a kind of transit's routes data processing method based on big data, it is characterised in that including：
Server cluster obtains the business datum of each track circuit in Metro Network from data collection layer；
The server cluster is handled and stored to the business datum of acquisition；
The server cluster is transported according to the business datum after processing to the traffic of each track circuit in the Metro Network
Row state is controlled.
2. the method as described in claim 1, it is characterised in that the server cluster includes data acquisition server and interface
Server；
The server cluster obtains the business datum of each track circuit in Metro Network from data collection layer, including：
The server cluster obtains the business datum of each track circuit by the interface server from data collection layer, described
The business datum of each track circuit is to be gathered by the data acquisition server from the service sub-system of each track circuit
And be stored in data collection layer.
3. the method as described in claim 1, it is characterised in that the server cluster includes data processing server and storage
Device；
The server cluster is handled and stored to the business datum of acquisition, including：
The data processing server carries out data pick-up and data cleansing to the business datum of collection；
Business datum after cleaning is carried out data conversion by the data processing server；
The data processing server preserves the business datum after data conversion into the memory.
4. method as claimed in claim 3, it is characterised in that the memory includes real-time memory and distributed storage
Device；
The data processing server preserves the business datum after data conversion into the memory, including：
When the business datum is the data for needing real-time release, by the data for needing real-time release be stored in it is described in real time
In memory；
After it is determined that the data for needing real-time release are issued, the data for needing real-time release are preserved to institute
State in distributed memory；
The business datum preserves the data for not needing real-time release to described point not need during real-time release data
In cloth memory.
5. method as claimed in claim 4, it is characterised in that the server cluster includes data analytics server and control
Central server；
The server cluster is transported according to the business datum after processing to the traffic of each track circuit in the Metro Network
Row state is controlled, including：
The data analytics server sets up analysis model by visual logical edit instrument；
The data analytics server carries out data according to the analysis model to the business datum in the distributed memory
Analysis；
Traffic of the control centre's server based on data analysis result to each track circuit in the Metro Network is transported
Row state is controlled.
6. a kind of server cluster, it is characterised in that including：
Acquisition module, the business datum for obtaining each track circuit in Metro Network from data collection layer；
Processing module, is handled and is stored for the business datum to acquisition；And according to the business datum after processing to institute
The traffic circulation state for stating each track circuit in Metro Network is controlled.
7. server cluster as claimed in claim 6, it is characterised in that the acquisition module include data acquisition server and
Interface server；
The acquisition module specifically for：
The business datum of each track circuit, each described railway line are obtained from data collection layer by the interface server
The business datum on road is to gather and be stored in number from the service sub-system of each track circuit by the data acquisition server
According in acquisition layer.
8. server cluster as claimed in claim 6, it is characterised in that the processing module include data processing server and
Memory；
The processing module specifically for：
Data pick-up and data cleansing are carried out to the business datum of collection by the data processing server；
Business datum after cleaning is carried out by data conversion by the data processing server；
The business datum after data conversion is preserved into the memory by the data processing server.
9. server cluster as claimed in claim 8, it is characterised in that the memory includes real-time memory and distribution
Memory；
The processing module specifically for：
When the business datum is the data for needing real-time release, by the data for needing real-time release be stored in it is described in real time
In memory；
After it is determined that the data for needing real-time release are issued, the data for needing real-time release are preserved to institute
State in distributed memory；
The business datum preserves the data for not needing real-time release to described point not need during real-time release data
In cloth memory.
10. server cluster as claimed in claim 9, it is characterised in that the processing module includes data analytics server
With control centre's server；
The processing module specifically for：
The data analytics server sets up analysis model by visual logical edit instrument；
The data analytics server carries out data according to the analysis model to the business datum in the distributed memory
Analysis；
Traffic of the control centre's server based on data analysis result to each track circuit in the Metro Network is transported
Row state is controlled.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710225226.5A CN107133273A (en) | 2017-04-07 | 2017-04-07 | A kind of transit's routes data processing method and server cluster based on big data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710225226.5A CN107133273A (en) | 2017-04-07 | 2017-04-07 | A kind of transit's routes data processing method and server cluster based on big data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107133273A true CN107133273A (en) | 2017-09-05 |
Family
ID=59716694
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710225226.5A Pending CN107133273A (en) | 2017-04-07 | 2017-04-07 | A kind of transit's routes data processing method and server cluster based on big data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107133273A (en) |
Cited By (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108009290A (en) * | 2017-12-25 | 2018-05-08 | 国电南瑞科技股份有限公司 | A kind of data modeling and storage method of track traffic command centre gauze big data |
| CN108415944A (en) * | 2018-01-30 | 2018-08-17 | 长安大学 | Real time computation system and its implementation based on micro services under a kind of traffic environment |
| CN109558417A (en) * | 2018-11-28 | 2019-04-02 | 亚信科技（南京）有限公司 | A kind of data processing method and platform |
| CN109741106A (en) * | 2018-12-29 | 2019-05-10 | 鼎信信息科技有限责任公司 | Data processing method, device, system, computer equipment and readable storage medium storing program for executing |
| CN110110170A (en) * | 2019-04-30 | 2019-08-09 | 北京字节跳动网络技术有限公司 | A kind of method, apparatus of data processing, medium and electronic equipment |
| CN110445648A (en) * | 2019-08-05 | 2019-11-12 | 卡斯柯信号有限公司 | A kind of O&M device for super-large city Metro Network grade signal of communication |
| CN110851523A (en) * | 2019-10-29 | 2020-02-28 | 交控科技股份有限公司 | Line network scheduling system |
| CN110971687A (en) * | 2019-11-29 | 2020-04-07 | 浙江邦盛科技有限公司 | Rail transit flow data processing method |
| CN111114597A (en) * | 2019-12-28 | 2020-05-08 | 卡斯柯信号有限公司 | Train tracking processing method of COCC automatic monitoring system |
| CN111753034A (en) * | 2020-05-20 | 2020-10-09 | 广东省国土资源测绘院 | One-stop type geographical big data platform |
| CN112186803A (en) * | 2020-09-29 | 2021-01-05 | 国网上海市电力公司 | Big data platform system of distribution network |
| CN112650897A (en) * | 2020-12-23 | 2021-04-13 | 广西交控智维科技发展有限公司 | Information monitoring system and display method based on track asset system |
| CN112693502A (en) * | 2019-10-23 | 2021-04-23 | 上海宝信软件股份有限公司 | Urban rail transit monitoring system and method based on big data architecture |
| CN112765259A (en) * | 2021-01-20 | 2021-05-07 | 青岛海信网络科技股份有限公司 | Data processing method and device for subway line network center |
| CN113656011A (en) * | 2021-08-17 | 2021-11-16 | 广州新科佳都科技有限公司 | Visual development system of track traffic net low code |
| CN113704310A (en) * | 2021-09-08 | 2021-11-26 | 北京城建设计发展集团股份有限公司 | Rail transit driving command system and driving data processing method |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6920498B1 (en) * | 2000-08-31 | 2005-07-19 | Cisco Technology, Inc. | Phased learning approach to determining closest content serving sites |
| CN102495896A (en) * | 2011-12-13 | 2012-06-13 | 南京恩瑞特实业有限公司 | Method for implementing automatic generation tool of track line database |
| CN103391185A (en) * | 2013-08-12 | 2013-11-13 | 北京泰乐德信息技术有限公司 | Cloud security storage and processing method and system for rail transit monitoring data |
| CN103457816A (en) * | 2013-08-29 | 2013-12-18 | 上海铁路通信有限公司 | Jumper connection looped network system applicable to rail transit vehicle |
| CN105204501A (en) * | 2015-09-28 | 2015-12-30 | 邹海英 | High-speed unmanned vehicle and system based on electronic track |
-
2017
- 2017-04-07 CN CN201710225226.5A patent/CN107133273A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6920498B1 (en) * | 2000-08-31 | 2005-07-19 | Cisco Technology, Inc. | Phased learning approach to determining closest content serving sites |
| CN102495896A (en) * | 2011-12-13 | 2012-06-13 | 南京恩瑞特实业有限公司 | Method for implementing automatic generation tool of track line database |
| CN103391185A (en) * | 2013-08-12 | 2013-11-13 | 北京泰乐德信息技术有限公司 | Cloud security storage and processing method and system for rail transit monitoring data |
| CN103457816A (en) * | 2013-08-29 | 2013-12-18 | 上海铁路通信有限公司 | Jumper connection looped network system applicable to rail transit vehicle |
| CN105204501A (en) * | 2015-09-28 | 2015-12-30 | 邹海英 | High-speed unmanned vehicle and system based on electronic track |
Cited By (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108009290A (en) * | 2017-12-25 | 2018-05-08 | 国电南瑞科技股份有限公司 | A kind of data modeling and storage method of track traffic command centre gauze big data |
| CN108009290B (en) * | 2017-12-25 | 2022-03-15 | 国电南瑞科技股份有限公司 | Data modeling and storage method for large data of rail transit command center line network |
| CN108415944A (en) * | 2018-01-30 | 2018-08-17 | 长安大学 | Real time computation system and its implementation based on micro services under a kind of traffic environment |
| CN108415944B (en) * | 2018-01-30 | 2019-03-22 | 长安大学 | Real time computation system and its implementation based on micro services under a kind of traffic environment |
| CN109558417A (en) * | 2018-11-28 | 2019-04-02 | 亚信科技（南京）有限公司 | A kind of data processing method and platform |
| CN109558417B (en) * | 2018-11-28 | 2023-08-08 | 亚信科技（南京）有限公司 | Data processing method and system |
| CN109741106A (en) * | 2018-12-29 | 2019-05-10 | 鼎信信息科技有限责任公司 | Data processing method, device, system, computer equipment and readable storage medium storing program for executing |
| CN110110170A (en) * | 2019-04-30 | 2019-08-09 | 北京字节跳动网络技术有限公司 | A kind of method, apparatus of data processing, medium and electronic equipment |
| CN110445648A (en) * | 2019-08-05 | 2019-11-12 | 卡斯柯信号有限公司 | A kind of O&M device for super-large city Metro Network grade signal of communication |
| CN112693502A (en) * | 2019-10-23 | 2021-04-23 | 上海宝信软件股份有限公司 | Urban rail transit monitoring system and method based on big data architecture |
| CN110851523A (en) * | 2019-10-29 | 2020-02-28 | 交控科技股份有限公司 | Line network scheduling system |
| CN110971687A (en) * | 2019-11-29 | 2020-04-07 | 浙江邦盛科技有限公司 | Rail transit flow data processing method |
| CN111114597A (en) * | 2019-12-28 | 2020-05-08 | 卡斯柯信号有限公司 | Train tracking processing method of COCC automatic monitoring system |
| CN111753034A (en) * | 2020-05-20 | 2020-10-09 | 广东省国土资源测绘院 | One-stop type geographical big data platform |
| CN112186803A (en) * | 2020-09-29 | 2021-01-05 | 国网上海市电力公司 | Big data platform system of distribution network |
| CN112650897A (en) * | 2020-12-23 | 2021-04-13 | 广西交控智维科技发展有限公司 | Information monitoring system and display method based on track asset system |
| CN112765259A (en) * | 2021-01-20 | 2021-05-07 | 青岛海信网络科技股份有限公司 | Data processing method and device for subway line network center |
| CN113656011A (en) * | 2021-08-17 | 2021-11-16 | 广州新科佳都科技有限公司 | Visual development system of track traffic net low code |
| CN113656011B (en) * | 2021-08-17 | 2022-03-11 | 广州新科佳都科技有限公司 | Visual development system of track traffic net low code |
| CN113704310A (en) * | 2021-09-08 | 2021-11-26 | 北京城建设计发展集团股份有限公司 | Rail transit driving command system and driving data processing method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107133273A (en) | A kind of transit's routes data processing method and server cluster based on big data | |
| CN103500173B (en) | A kind of querying method of track traffic Monitoring Data | |
| CN104618693B (en) | A kind of monitor video based on cloud computing handles task management method and system online | |
| CN104462121B (en) | Data processing method, apparatus and system | |
| CN107302466A (en) | A kind of power ＆ environment supervision system big data analysis platform and method | |
| CN107729413A (en) | Regional traffic intelligent management system based on big data | |
| CN106973119A (en) | A kind of electric power enterprise storage resource management system | |
| CN110351150A (en) | Fault rootstock determines method and device, electronic equipment and readable storage medium storing program for executing | |
| CN107894990A (en) | A kind of city general utility functions platform | |
| CN105871957B (en) | Monitoring framework design method and monitoring server, agent unit, control server | |
| CN105427221A (en) | Cloud platform-based police affair management method | |
| Zhang et al. | A deep-intelligence framework for online video processing | |
| CN104616205A (en) | Distributed log analysis based operation state monitoring method of power system | |
| CN105681768A (en) | Method of realizing real-time people stream monitoring through communication data | |
| CN106210124B (en) | A kind of unified cloud data center monitoring system | |
| CN110458350A (en) | Infrastructure service platform construction method, device and the electronic equipment of railway traffic system | |
| CN112698953A (en) | Power grid intelligent operation and detection platform based on micro-service | |
| CN103546313A (en) | Cloud computing based IT (information technology) operation and maintenance management system | |
| CN109167689A (en) | network device monitoring method, device and server | |
| CN102932448A (en) | Distributed network crawler URL (uniform resource locator) duplicate removal system and method | |
| CN107968482A (en) | A kind of generation of electricity by new energy station management platform | |
| CN106777079A (en) | A kind of daily record data Visualized Analysis System and method | |
| CN107729219A (en) | Resource monitoring method, device and terminal based on super fusion storage system | |
| CN106534784A (en) | Acquisition analysis storage statistical system for video analysis data result set | |
| CN109739820A (en) | A kind of E-government information service system based on big data analysis |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170905 |