CN109634998A - A kind of traffic journey characteristic analysis platform based on mobile phone signaling big data - Google Patents

A kind of traffic journey characteristic analysis platform based on mobile phone signaling big data Download PDF

Info

Publication number
CN109634998A
CN109634998A CN201811373224.1A CN201811373224A CN109634998A CN 109634998 A CN109634998 A CN 109634998A CN 201811373224 A CN201811373224 A CN 201811373224A CN 109634998 A CN109634998 A CN 109634998A
Authority
CN
China
Prior art keywords
data
module
mobile phone
traffic
result
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201811373224.1A
Other languages
Chinese (zh)
Inventor
方秀川
陈智宏
翁剑成
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Thoroughfare Permanent Technology Co Ltd
Original Assignee
Beijing Thoroughfare Permanent Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Thoroughfare Permanent Technology Co Ltd filed Critical Beijing Thoroughfare Permanent Technology Co Ltd
Priority to CN201811373224.1A priority Critical patent/CN109634998A/en
Publication of CN109634998A publication Critical patent/CN109634998A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q50/00Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
    • G06Q50/10Services
    • G06Q50/26Government or public services
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/02Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]

Landscapes

  • Business, Economics & Management (AREA)
  • Tourism & Hospitality (AREA)
  • Engineering & Computer Science (AREA)
  • Marketing (AREA)
  • Strategic Management (AREA)
  • Economics (AREA)
  • General Health & Medical Sciences (AREA)
  • Human Resources & Organizations (AREA)
  • Educational Administration (AREA)
  • Primary Health Care (AREA)
  • Health & Medical Sciences (AREA)
  • Physics & Mathematics (AREA)
  • General Business, Economics & Management (AREA)
  • General Physics & Mathematics (AREA)
  • Development Economics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Mobile Radio Communication Systems (AREA)

Abstract

The invention discloses a kind of traffic journey characteristic analysis platforms based on mobile phone signaling big data, including data preprocessing module, data mart modeling module, data distribution module, data acquisition module, data statistic analysis module and data to visualize module.Mobile phone signaling data is pre-processed by the big data analysis platform of operator, the relevant available fields of extraction platform, rejects noise;Pretreated signaling data is processed, carries out traffic model calculating using spark big data technology;The Trip chain result data that big data calculates is distributed in redis cluster;It is acquired by result of the http mode to data mart modeling by the security gateway of major operator, result data is stored on the server of user according to rule after acquisition;The population duty for collecting local result data and carrying out traffic service is lived and the analyses such as trip characteristics;Data visualization is carried out to traffic service analysis result, the component exhibiting of various configurations can be used.

Description

A kind of traffic journey characteristic analysis platform based on mobile phone signaling big data
Technical field
The present invention relates to Urban Traffic Plannings and big data processing technology field, especially a kind of to be based on the big number of mobile phone signaling According to traffic journey characteristic analysis platform.
Background technique
With the rapid development of social economy with the continuous improvement of the national level of urbanization, urban construction constantly improve, To Urban Traffic Planning, construction and management, more stringent requirements are proposed.For Urban Planner, with information technology, lead to Letter technology and mobile phone terminal it is universal, be gradually taken seriously using the new technologies acquisition traffic information such as mobile phone positioning, huge hand Machine user group provides a large amount of data source for the acquisition of traffic data, for the analysis of resident's traffic journey characteristic provide it is abundant can The data leaned on are supported.
Mobile phone signaling data be mobile phone user make a phone call, send short messages, change in location and while periodically updating generate Mobile position data, recently as the development of wireless location technology, mobile phone signaling data constantly improve and increases, so that utilizing Mobile phone positioning come calculate urban population duty live, trip characteristics, trip distance, Commuting Distance on and off duty become a kind of possibility.This phase For conventional survey mode, it is available more comprehensively, more acurrate, more real-time data, provided for traffic journey characteristic analysis Good data basis.For example, big mobile communication carrier, Beijing three (China Mobile, China Unicom and China Telecom) is daily Signaling data about 6000G of 2G, 3G and 4G of generation or so, data set huge in this way, need big data technology come into Row storage and processing.
Summary of the invention
It can be gone out by the progress traffic of the mobile phone signaling big data of mobile operator the purpose of the present invention is to provide a kind of Row signature analysis provides Data safeguard for the traffic programme in city.
To achieve the above object, the invention adopts the following technical scheme: a kind of traffic based on mobile phone signaling big data Trip characteristics analysis platform, including data preprocessing module, data mart modeling module, data distribution module, data acquisition module, number Analysis module and data visualize module according to statistics.Data acquisition module, data statistic analysis module and data visualization Display module is sequentially connected with, and data preprocessing module, data mart modeling module and data distribution module are sequentially connected with, data distribution mould It is connected between block and data acquisition module by security gateway, this six functions module is combined closely, pre- from the data of data source The data of processing to the end are visualized, this data preprocessing module, data mart modeling module, data distribution module, number According to acquisition module, the traffic journey characteristic of data statistic analysis module five functional module composition based on mobile phone signaling big data The nucleus module of analysis platform enables to the platform can flexible expansion, stabilization, efficient operation.
A kind of traffic journey characteristic analysis platform based on mobile phone signaling big data, each functional module for including is specifically such as Under:
Data preprocessing module pre-processes the mobile phone signaling data of operator, extracts useful field, composition with User imei code (DecryptDecryption) is the position coordinate data of critical field, and carries out abnormal judgement according to speed and angle, will be different Normal point is removed.
Data mart modeling module is clustered to pretreated signaling data had been carried out using the clustering algorithm of optimization Analysis forms accumulation point, uses the space length algorithm after optimization simultaneously in clustering algorithm, picks out each mobile phone user The daily all dwell points to sort according to chronological order and stop beginning and ending time, form the traffic trip of the mobile phone user Chain, and be stored in operator's big data platform HDFS file system.
Data distribution module, to traffic trip chain result data, data volume is big, by spark big data computing engines into Result data is distributed in the redis cluster of operator by row data distribution, while the total amount of data size of this distribution being sent out Cloth is into redis cluster, so that data volume of the data acquisition module to acquisition is effectively verified.
Data acquisition module, operator external user send get or post request by https mode and penetrate operator Security gateway obtains trip link by the key value being previously set in http request parameter from the redis cluster of operator Collected result data is uniformly stored in use according to the value after rear two hash of the imei code of mobile phone user by fruit data In 256 data files in the presence server of family, subsequent traffic service is facilitated to statistically analyze.
Data statistic analysis module, the intermediate result Trip chain data arrived using data collecting module collected, according to design Unified traffic model analysis interface, these traffic service models contain urban population duty live, population OD, trip distance, Analysis models, these analysis models such as commuter distance, trip number are configured in platform with plug-in mode, are provided for user Flexibly, easily business diagnosis, analysis result are stored in relational database, and graphical data is facilitated to show.
Data visualization module, according to data statistic analysis as a result, Service Component pair can be selected flexibly from platform Data are visualized, these Service Component contain GIS map, thermodynamic chart, line chart, histogram, instrument board etc., Neng Gouzhi Xml configuration mode was connected, it is convenient and practical.
Detailed description of the invention
Fig. 1 is the structural block diagram of this system.
Fig. 2 is the mobile original mobile phone signaling sample data of operator in Beijing.
Fig. 3 is the original mobile phone signaling sample data of operator of Beijing connection.
Fig. 4 is the original mobile phone signaling sample data of operator of Beijing Telecom.
Fig. 5 is Trip chain result data sample.
Specific embodiment
A kind of traffic journey characteristic analysis platform based on mobile phone signaling big data, including data prediction, data mart modeling, Data distribution, data acquisition, data statistic analysis and data visualization etc..
As shown in Figure 1, the mobile phone signaling data amount of operator is huge, day regular data increment it is swift and violent, be stored in major fortune It seeks in the hdfs file system of the big data platform of quotient, signaling data must be processed by existing big data analysis technology Processing, the platform that the present invention designs use spark technology, and spark is to aim at large-scale data processing and design quick logical Memory computing engines.
Its described data preprocessing module, due to operator provide the original not all data of mobile phone signaling data all Be it is qualified, than if any lack longitude and latitude data, some lacks user's unique identification IMSI code, and some latitude and longitude coordinates are got over Boundary, then Data duplication etc., pretreated result are then that one day data of a user are weeded out hashed field to some, are merged into Together, and according to time-sequencing.The original mobile phone signaling sample data of movement in the big operator in Beijing three is as shown in Figure 2:
Explanation of field is as follows:
.The original mobile phone signaling sample data of connection in the big operator in Beijing three is as shown in Figure 3:
Explanation of field is as follows:
The original mobile phone signaling sample data of movement in the big operator in Beijing three is as shown in Figure 4:
Explanation of field is as follows:
Data mart modeling module described in it improves arithmetic speed in order to utmostly reduce operand, uses excellent Clustering algorithm after change.When being clustered to the data of each user, all points of the user are put together space clustering, it can Multiple clusters can be clustered into, when generating final Trip chain data, will also actually be used according to time continuity sub-clustering again User data is clustered in two dimensions of room and time.
In the case where this dual-dimension cluster, there is very efficient optimal way in clustering algorithm.Due to number of users According to being sorted in advance according to the time, the method discontinuously clustered can be taken, while carrying out room and time cluster.It is specific to calculate Method is as follows:
First point is put into cluster, subsequent point singly judges whether in cluster, if be added in cluster Cluster, if it was not then all the points room and time cluster of front is is completed, this point is placed in a new cluster, after continuation Continuous cluster, and so on.When actually using the clustering algorithm after this optimization, calculating speed is substantially increased, is saved Computing resource.When clustering algorithm carries out space clustering calculating, using the method for calculating two o'clock longitude and latitude difference, according to estimating It calculates, if dimension is identical, the difference of the longitude of two coordinate points is greater than 0.0036, then more than 300 meters of cluster radius;If longitude phase Together, the difference of longitude is greater than 0.0027, then more than 300 meters of cluster radius, need to only calculate the difference of the longitude and latitude of two o'clock in this way, so that it may Judge space length, provides computational efficiency significantly.And traditional normed space distance algorithm contain addition subtraction multiplication and division, sine more than The various operations such as string, algorithm comparison is complicated, in data cleansing and clustering algorithm, will calculate two o'clock space length, is counting greatly In the case where amount, efficiency is extremely low.
Trip chain result data sample is as shown in Figure 5.
Explanation of field:
IMSI: time started, end time, longitude, latitude, cluster points;Time started, end time, longitude, latitude Degree, cluster points ...
Data distribution module described in it, due to that need to pass the Trip chain result of calculating from the big data platform of operator back On the server of data consumer, it will calculate in the Trip chain data distribution to the redis cluster of operator completed, user can It is directly acquired with passing through the security gateway of operator from redis cluster.
Data acquisition module described in it, data consumer are sent get/post request, are passed through by http request mode The security gateway acquisition Trip chain data of operator are simultaneously saved on the server of user local.Since Trip chain data are larger (about 20G daily), data acquisition module use multi-process multithreading, collected Trip chain data are stored in service In 256 files on device.To improve treatment effeciency, this acquisition module realizes that multi-process acquires using Python, at acquisition Reason process is as follows:
1. newly-built 0-f totally 16 files create 0-f totally 16 files under each file;
2. each row of data obtained by http request, carries out md5 encryption for IMSI, according to encrypted field last two Position, is written in corresponding 0-f, 0-f file, such as is d1ff16c637c9c859c545e66ccdb5acc8 after encryption, then will 8 files under c catalogue are written in this row data.
Data statistic analysis module described in it is realized by the Callable multithreading of Java, is patrolled previously according to server It collects CPU number and generates thread pool, then analyze processing respectively according to traffic service model algorithm by the thread in thread pool by counting According to collected 256 data files of acquisition module, the analyses such as residence, employment ground, commuting OD are obtained as a result, and protecting result It is stored in relevant database.
Data visualization module described in it, to the analysis generated by statistical analysis module as a result, passing through visualization component Flexibly shown.These components contain GIS map, thermodynamic chart, line chart, histogram, instrument board etc., can be matched by xml Parameter is set, browser directly parses.

Claims (7)

1. a kind of traffic journey characteristic analysis platform based on mobile phone signaling big data, it is characterised in that: including data prediction Module, data mart modeling module, data distribution module, data acquisition module, data statistic analysis module and data visualize Module;Data acquisition module, data statistic analysis module and data visualize module and are sequentially connected with, data prediction mould Block, data mart modeling module and data distribution module are sequentially connected with, and pass through safety between data distribution module and data acquisition module Gateway connection, this six functions module combines closely, and carries out visualization exhibition from the data of the data prediction of data source to the end Show, this data preprocessing module, data mart modeling module, data distribution module, data acquisition module, data statistic analysis module five Big functional module constitutes the nucleus module of the traffic journey characteristic analysis platform based on mobile phone signaling big data.
2. a kind of traffic journey characteristic analysis platform based on mobile phone signaling big data according to claim 1, feature It is:
Data preprocessing module pre-processes the mobile phone signaling data of operator, extracts useful field, forms with user Imei code (DecryptDecryption) is the position coordinate data of critical field, and carries out abnormal judgement according to speed and angle, by abnormal point Remove position.
3. a kind of traffic journey characteristic analysis platform based on mobile phone signaling big data according to claim 1, feature Be: data mart modeling module carried out clustering using the clustering algorithm of optimization to pretreated signaling data has been carried out Accumulation point is formed, the space length algorithm after optimization is used simultaneously in clustering algorithm, it is daily to pick out each mobile phone user According to chronological order sequence all dwell points and stop the beginning and ending time, form the traffic trip chain of the mobile phone user, And it is stored in operator's big data platform HDFS file system.
4. a kind of traffic journey characteristic analysis platform based on mobile phone signaling big data according to claim 1, feature Be: data distribution module, to traffic trip chain result data, data volume is big, is counted by spark big data computing engines According to distribution, result data is distributed in the redis cluster of operator, while this total amount of data size distributed being published to In redis cluster, so that data volume of the data acquisition module to acquisition is effectively verified.
5. a kind of traffic journey characteristic analysis platform based on mobile phone signaling big data according to claim 1, feature Be: data acquisition module, operator external user send the peace that get or post request penetrates operator by https mode Full gateway obtains Trip chain result by the key value being previously set in http request parameter from the redis cluster of operator Collected result data is uniformly stored in user according to the value after rear two hash of the imei code of mobile phone user by data In 256 data files in presence server, subsequent traffic service is facilitated to statistically analyze.
6. a kind of traffic journey characteristic analysis platform based on mobile phone signaling big data according to claim 1, feature It is: data statistic analysis module, the intermediate result Trip chain data arrived using data collecting module collected, according to the system of design One traffic model analysis interface, these traffic service models contain urban population duty live, population OD, trip distance, commuting Analysis models, these analysis models such as trip distance, trip number are configured in platform with plug-in mode, provide spirit for user Living, convenient and fast business diagnosis, analysis result are stored in relational database, and graphical data is facilitated to show.
7. a kind of traffic journey characteristic analysis platform based on mobile phone signaling big data according to claim 1, feature It is: data visualization module, according to data statistic analysis as a result, selecting Service Component to carry out visually data from platform Change, these Service Component contain GIS map, thermodynamic chart, line chart, histogram, instrument board etc., can directly be matched by xml Set mode.
CN201811373224.1A 2018-11-19 2018-11-19 A kind of traffic journey characteristic analysis platform based on mobile phone signaling big data Pending CN109634998A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201811373224.1A CN109634998A (en) 2018-11-19 2018-11-19 A kind of traffic journey characteristic analysis platform based on mobile phone signaling big data

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201811373224.1A CN109634998A (en) 2018-11-19 2018-11-19 A kind of traffic journey characteristic analysis platform based on mobile phone signaling big data

Publications (1)

Publication Number Publication Date
CN109634998A true CN109634998A (en) 2019-04-16

Family

ID=66068491

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201811373224.1A Pending CN109634998A (en) 2018-11-19 2018-11-19 A kind of traffic journey characteristic analysis platform based on mobile phone signaling big data

Country Status (1)

Country Link
CN (1) CN109634998A (en)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110807546A (en) * 2019-10-22 2020-02-18 恒大智慧科技有限公司 Community grid population change early warning method and system
CN110807547A (en) * 2019-10-22 2020-02-18 恒大智慧科技有限公司 Method and system for predicting family population structure
CN112183904A (en) * 2020-11-19 2021-01-05 北京清研宏达信息科技有限公司 Bus route optimization method based on resident travel OD
CN112231392A (en) * 2020-10-29 2021-01-15 广东机场白云信息科技有限公司 Civil aviation customer source data analysis method, electronic equipment and computer readable storage medium
CN112256752A (en) * 2020-10-13 2021-01-22 山东三木众合信息科技股份有限公司 Data prediction processing method based on data mining
CN116206452A (en) * 2023-05-04 2023-06-02 北京城建交通设计研究院有限公司 Sparse data characteristic analysis method and system for urban traffic travel

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105142106A (en) * 2015-07-29 2015-12-09 西南交通大学 Traveler home-work location identification and trip chain depicting method based on mobile phone signaling data
CN107040894A (en) * 2017-04-21 2017-08-11 杭州市综合交通研究中心 A kind of resident trip OD acquisition methods based on mobile phone signaling data
CN107305590A (en) * 2017-06-14 2017-10-31 北京市交通信息中心 A kind of urban transportation trip characteristicses based on mobile phone signaling data determine method
CN108197261A (en) * 2017-12-30 2018-06-22 北京通途永久科技有限公司 A kind of wisdom traffic operating system
CN108650632A (en) * 2018-04-28 2018-10-12 广州市交通规划研究院 It is a kind of based on duty live correspondence and when space kernel clustering stationary point judgment method

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105142106A (en) * 2015-07-29 2015-12-09 西南交通大学 Traveler home-work location identification and trip chain depicting method based on mobile phone signaling data
CN107040894A (en) * 2017-04-21 2017-08-11 杭州市综合交通研究中心 A kind of resident trip OD acquisition methods based on mobile phone signaling data
CN107305590A (en) * 2017-06-14 2017-10-31 北京市交通信息中心 A kind of urban transportation trip characteristicses based on mobile phone signaling data determine method
CN108197261A (en) * 2017-12-30 2018-06-22 北京通途永久科技有限公司 A kind of wisdom traffic operating system
CN108650632A (en) * 2018-04-28 2018-10-12 广州市交通规划研究院 It is a kind of based on duty live correspondence and when space kernel clustering stationary point judgment method

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
李振邦 等: "基于数据挖掘的手机用户出行方式识别研究", 《黑龙江科技信息》 *

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110807546A (en) * 2019-10-22 2020-02-18 恒大智慧科技有限公司 Community grid population change early warning method and system
CN110807547A (en) * 2019-10-22 2020-02-18 恒大智慧科技有限公司 Method and system for predicting family population structure
CN112256752A (en) * 2020-10-13 2021-01-22 山东三木众合信息科技股份有限公司 Data prediction processing method based on data mining
CN112231392A (en) * 2020-10-29 2021-01-15 广东机场白云信息科技有限公司 Civil aviation customer source data analysis method, electronic equipment and computer readable storage medium
CN112231392B (en) * 2020-10-29 2024-09-06 广东机场白云信息科技股份有限公司 Civil aviation passenger source data analysis method, electronic equipment and computer readable storage medium
CN112183904A (en) * 2020-11-19 2021-01-05 北京清研宏达信息科技有限公司 Bus route optimization method based on resident travel OD
CN116206452A (en) * 2023-05-04 2023-06-02 北京城建交通设计研究院有限公司 Sparse data characteristic analysis method and system for urban traffic travel
CN116206452B (en) * 2023-05-04 2023-08-15 北京城建交通设计研究院有限公司 Sparse data characteristic analysis method and system for urban traffic travel

Similar Documents

Publication Publication Date Title
CN109634998A (en) A kind of traffic journey characteristic analysis platform based on mobile phone signaling big data
CN105069703B (en) A kind of electrical network mass data management method
CN103838863B (en) A kind of big data clustering algorithm based on cloud computing platform
CN107590250A (en) A kind of space-time orbit generation method and device
CN106960006A (en) Measuring similarity system and its measure between a kind of different tracks
CN106778876A (en) User classification method and system based on mobile subscriber track similitude
CN111352808B (en) Alarm data processing method, device, equipment and storage medium
CN111930868A (en) Big data behavior trajectory analysis method based on multi-dimensional data acquisition
CN109254901B (en) A kind of Monitoring Indexes method and system
CN104035954A (en) Hadoop-based recognition method for fake-licensed car
CN117971606B (en) Log management system and method based on elastic search
CN114550076A (en) Method, device and equipment for monitoring area abnormal behaviors and storage medium
CN113032403B (en) Data insight method, device, electronic equipment and storage medium
WO2021114615A1 (en) Method, apparatus, and device for visualization of behavior risk identification, and storage medium
Ding et al. Massive heterogeneous sensor data management in the Internet of Things
CN113704567A (en) Internet of things data management method and device, Internet of things data resource pool and equipment
CN114418360A (en) Smart city operation sign big data analysis method and device
WO2022116326A1 (en) Transportation information processing method, device, terminal, and computer-readable storage medium
CN102081641A (en) Design of GPU-based real-time data stream high-speed matching analysis system
CN104778355A (en) Trajectory outlier detection method based on wide-area distributed traffic system
CN113157521A (en) Monitoring method and monitoring system for whole life cycle of block chain
CN117951341A (en) Block chain-based service processing method, device, equipment, medium and product
CN104361138B (en) Cost sharing and apparatus and method of tracing to the source
CN109165203A (en) Large public building energy consumption data based on Hadoop framework stores analysis method
CN109687592A (en) Substation information display control method, device and electronic equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication

Application publication date: 20190416

RJ01 Rejection of invention patent application after publication