CN116665441A - Intelligent traffic monitoring analysis system based on big data technology - Google Patents

Intelligent traffic monitoring analysis system based on big data technology Download PDF

Info

Publication number
CN116665441A
CN116665441A CN202310617204.9A CN202310617204A CN116665441A CN 116665441 A CN116665441 A CN 116665441A CN 202310617204 A CN202310617204 A CN 202310617204A CN 116665441 A CN116665441 A CN 116665441A
Authority
CN
China
Prior art keywords
data
model
analysis
traffic
road
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310617204.9A
Other languages
Chinese (zh)
Inventor
刘丽娟
吕科
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Dalian Jiaotong University
Original Assignee
Dalian Jiaotong University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Dalian Jiaotong University filed Critical Dalian Jiaotong University
Priority to CN202310617204.9A priority Critical patent/CN116665441A/en
Publication of CN116665441A publication Critical patent/CN116665441A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21Design, administration or maintenance of databases
    • G06F16/215Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22Indexing; Data structures therefor; Storage structures
    • G06F16/2282Tablespace storage structures; Management thereof
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/242Query formulation
    • G06F16/2433Query languages
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/28Databases characterised by their database models, e.g. relational or object models
    • G06F16/284Relational databases
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00Machine learning
    • G06N20/20Ensemble learning
    • GPHYSICS
    • G08SIGNALLING
    • G08GTRAFFIC CONTROL SYSTEMS
    • G08G1/00Traffic control systems for road vehicles
    • G08G1/01Detecting movement of traffic to be counted or controlled
    • G08G1/0104Measuring and analyzing of parameters relative to traffic conditions
    • G08G1/0125Traffic data processing
    • GPHYSICS
    • G08SIGNALLING
    • G08GTRAFFIC CONTROL SYSTEMS
    • G08G1/00Traffic control systems for road vehicles
    • G08G1/01Detecting movement of traffic to be counted or controlled
    • G08G1/0104Measuring and analyzing of parameters relative to traffic conditions
    • G08G1/0137Measuring and analyzing of parameters relative to traffic conditions for specific applications
    • GPHYSICS
    • G08SIGNALLING
    • G08GTRAFFIC CONTROL SYSTEMS
    • G08G1/00Traffic control systems for road vehicles
    • G08G1/065Traffic control systems for road vehicles by counting the vehicles in a section of the road or in a parking area, i.e. comparing incoming count with outgoing count

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Databases & Information Systems (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Chemical & Material Sciences (AREA)
  • Analytical Chemistry (AREA)
  • Mathematical Physics (AREA)
  • Computing Systems (AREA)
  • Computational Linguistics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Computation (AREA)
  • Medical Informatics (AREA)
  • Artificial Intelligence (AREA)
  • Quality & Reliability (AREA)
  • Traffic Control Systems (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

A large data technology-based intelligent traffic monitoring analysis system uses a high-availability mass log acquisition tool Flume to acquire data, and the data is transmitted to a distributed processing engine Flink through a high-throughput distributed message queue tool Kafka to clean the technical route of the flow data so as to acquire and process the road traffic data in real time. And carrying out different operations on the data according to different traffic data types. Aiming at the road archive data, the vehicle flow data establishes an integrated learning model LightGBM, predicts the traffic jam condition of the road in a short period in the future, and visually displays the predicted road condition information on a page; and aiming at real-time vehicle driving data information and traffic accident data, the data are analyzed and displayed in real time by using Hive and Spark technologies. The invention has wide application prospect in the development process of smart cities, performs services such as fine management on vehicles, and solves the problems of traffic jam and traffic violation caused by the rapid development of socioeconomic.

Description

Intelligent traffic monitoring analysis system based on big data technology
Technical Field
The invention relates to an intelligent traffic monitoring and analyzing system, in particular to an intelligent traffic monitoring and analyzing system based on a big data technology. Belongs to the technical field of traffic monitoring analysis.
Background
Aiming at the problems of energy consumption, pollution and congestion brought by the transportation industry, the development of intelligent transportation is one of ideas for solving a plurality of problems. The intelligent traffic is taken as important content of intelligent city development, and can provide services such as road congestion analysis and judgment, road traffic real-time monitoring and analysis, accident emergency rescue service, fine management of vehicles and the like. The intelligent traffic technology can effectively reduce traffic accidents, and the death number caused by the traffic accidents can be reduced by 30-70% each year.
With the development of various big data technologies and components, a new thought is provided for solving the problems of traffic jam, traffic violations and the like caused by the rapid development of socioeconomic performance, and the intelligent traffic monitoring analysis scheme based on the big data technology is provided by the system. On the basis of a large amount of traffic data generated every day, the problems of traffic illegal behaviors, road accident monitoring, road flow analysis, judgment, prediction, emergency rescue of accidents and the like are processed and solved in real time, and the development process of the smart city is quickened.
Disclosure of Invention
The intelligent traffic monitoring analysis system based on the big data technology aims to process and solve the problems of traffic illegal behaviors, road accident monitoring, road flow analysis, judgment prediction, accident emergency rescue and the like in real time on the basis of a large amount of traffic data generated every day, and quicken the development process of intelligent cities.
The technical solution of the invention is realized as follows:
an intelligent traffic monitoring analysis system based on big data technology, comprising: the system comprises a login module, a verification module, a road condition overview module, a congestion prediction module and a safety management module, and is characterized in that the login module inputs an account number and a password to enter the verification module after a user inputs a website to enter a platform; the back end acquires the account password, enters information and database account password information into a database for verification, returns an input account and password if the verification is successful, and inputs the account password again by a user, if the verification is successful, successfully enters a platform; the road condition overview module consists of data processing, data partition storage and data HQL analysis, and the data processing is used for road condition data positions
After the processing, the data is stored in a partitioning mode, a partitioning table is built when the Hive is used for data storage, optimization is carried out at the same time, the parquet format is used for storage, the snappy is used for data compression format, a large amount of data can be processed quickly, the data HQL analysis is carried out for data analysis, the data stored in the Hive is analyzed and calculated through an HSQL statement, and when the HQL statement cannot meet the requirement of the data to be processed, a user-defined UDF function is used for processing; the congestion prediction module consists of data acquisition, model prediction and data importing MySQL, the data acquisition acquires road congestion data, the model prediction performs data analysis, and a lightGBM model is innovatively used, so that the congestion prediction module is an efficient gradient lifting decision tree framework, and the application process in traffic flow prediction is as follows: (1) Preparing a data set, namely preparing the data set required by traffic flow prediction, wherein the data set comprises influence factors of factors such as historical traffic flow, weather conditions, holidays and the like on the traffic flow, preprocessing the data set, including data cleaning, missing value filling and characteristic engineering, and receiving the real-time flow number of a kafka message queue by using spark; (2) Model training, namely training a lightGBM model, wherein the process comprises the steps of selecting characteristics, setting parameters, training the model and verifying, wherein parameter adjustment can be divided into two parts, the first part is to determine basic parameters of a tree model, the second part is to adjust the model training parameters, the basic parameters of the tree model comprise the number of leaf nodes and the learning rate, and the optimal parameter combination is found by adopting a grid searching and Bayesian optimizing method; (3) After model evaluation and model training are completed, the model performance is evaluated by using a cross verification method, the model prediction accuracy is verified to be more than 90%, and data is imported into a MySQL (structured query language) and then is imported into a platform; the safety management module consists of data preprocessing, data warehouse design and data HQL analysis, the data preprocessing preprocesses safety data, the data warehouse design stores traffic data into the data warehouse, and the data layering is creatively adopted in the storage process: (1) A data operation layer (ODS), wherein the data is consistent with the original data downloaded by the official network, the history data stored in the Hive is read-only, and a front-end user and a background system are provided for inquiring the data; (2) The data detail layer (DWD) cleans, standardizes and dimension degenerates the data of the ODS layer, and splits the original data table into a user information table, a vehicle data table, a violation data table and other smaller tables for preparing the next analysis operation; (3) A data summarizing layer (DWS), wherein the data of the data summarizing layer are calculated and summarized according to the analysis subject, the personal information of the user and the corresponding vehicle information data are required to be inquired, the two tables are combined, and a data aggregation is focused on a multi-bin model with better complex inquiry and processing performance; (4) The data application layer (ADS) is also called a data mart, stores data analysis results, can be used for visually displaying data, can be convenient for users to add, delete and examine specific data, lightens the burden of a data warehouse, and analyzes and processes safety data through data HQL analysis.
The advantages of the present invention are obvious compared with the prior art, mainly expressed in that:
1. the system provides information service for user travel, can obtain real-time road condition, future road condition and current traffic event information, utilizes a big data platform analysis and judgment platform to carry out statistical analysis on high-speed urban traffic, and carries out service arrangement in advance in a targeted manner through the statistical analysis so as to ensure smooth road traffic.
2. Due to the lag of the vehicle management mode and the traffic information means, a large number of vehicles evade annual inspection and traffic insurance, motor vehicle violation event frequently occurs, vehicles escape after accident and the like, and the vehicles with abnormal phenomena are monitored in real time, so that traffic safety can be ensured, and the fine management of abnormal vehicles can be realized.
Drawings
The invention is shown in figure 5.
FIG. 1 is a schematic diagram of a system module of the present invention;
FIG. 2 is a schematic diagram of a login and authentication module of the present invention;
FIG. 3 is a schematic diagram of a road condition overview module according to the present invention;
FIG. 4 is a schematic diagram of a congestion prediction module according to the present invention
Fig. 5 is a schematic diagram of a security management module according to the present invention.
The system comprises a login module (101), a website input module (102), an account number and a password input module (2), a verification module (201), an account number and a password acquired by the rear end, a database input verification (202), a database input verification (203), a verification success (204), a platform input success (3), a road condition overview module (301) and a data processing; 302. storing the data in a partition mode; 303. data HQL analysis, 4, a congestion prediction module, 401, data acquisition, 402, model prediction, 403, data importing MySQL,5, a security management module, 501, data preprocessing, 502, data warehouse design, 503 and data HQL analysis.
Detailed Description
The intelligent traffic monitoring and analyzing system based on big data technology as shown in fig. 1, 2, 3, 4 and 5 comprises: the system is characterized in that the login module inputs an account number and a password 102 after a user inputs a website 101 to enter a platform, and enters the verification module 2; the back end obtains the account password 201, enters information and database account password information into a database for verification 202, returns an input account and password 102 if the verification is successful 203, and enters a platform 204 successfully if the verification is successful 203; the road condition overview module 3 is composed of data processing 301, data partition storage 302 and data HQL analysis 303, after the road condition data processing is carried out by the data processing 301, the data partition storage 302 carries out data storage, a partition table is built when the data storage is carried out by using Hive, optimization is carried out simultaneously, the partition table is stored by using parquet format, the compression format of the data is selected to be snappy, a large amount of data can be rapidly processed, the data HQL analysis 303 carries out data analysis, the data stored in the Hive is analyzed and calculated through HSQL sentences, and when the HQL sentences cannot meet the requirement of the data to be processed, the data is processed by using a custom UDF function; the congestion prediction module 4 consists of a data acquisition 401, a model prediction 402 and a data importing MySQL403, wherein the data acquisition 401 acquires road congestion data, the model prediction (402) performs data analysis, a lightGBM model is innovatively used, the model is an efficient gradient lifting decision tree framework, and the application process in traffic flow prediction is as follows: (1) Preparing a data set, namely preparing the data set required by traffic flow prediction, wherein the data set comprises influence factors of factors such as historical traffic flow, weather conditions, holidays and the like on the traffic flow, preprocessing the data set, including data cleaning, missing value filling and characteristic engineering, and receiving the real-time flow number of a kafka message queue by using spark; (2) Model training, namely training a lightGBM model, wherein the process comprises the steps of selecting characteristics, setting parameters, training the model and verifying, wherein parameter adjustment can be divided into two parts, the first part is to determine basic parameters of a tree model, the second part is to adjust the model training parameters, the basic parameters of the tree model comprise the number of leaf nodes and the learning rate, and the optimal parameter combination is found by adopting a grid searching and Bayesian optimizing method; (3) After model evaluation and model training are completed, the performance of the model is evaluated by using a cross-validation method, the prediction accuracy of the model is validated to be more than 90%, and data is imported into MySQL (403) to be imported into the platform 204; the safety management module 5 is composed of data preprocessing 501, data warehouse design 502 and data HQL analysis 503, wherein the data preprocessing 501 preprocesses safety data, the data warehouse design 502 stores traffic data into a data warehouse, and data layering is creatively adopted in the storage process: (1) A data operation layer (ODS), wherein the data is consistent with the original data downloaded by the official network, the history data stored in the Hive is read-only, and a front-end user and a background system are provided for inquiring the data; (2) The data detail layer (DWD) cleans, standardizes and dimension degenerates the data of the ODS layer, and splits the original data table into a user information table, a vehicle data table, a violation data table and other smaller tables for preparing the next analysis operation; (3) A data summarizing layer (DWS), wherein the data of the data summarizing layer are calculated and summarized according to the analysis subject, the personal information of the user and the corresponding vehicle information data are required to be inquired, the two tables are combined, and a data aggregation is focused on a multi-bin model with better complex inquiry and processing performance; (4) The data application layer (ADS), which is also called as data mart, stores data analysis results, can be used for visual display of data, can also facilitate users to add, delete and examine specific data, lighten the burden of a data warehouse, and analyze and process safety data by the data HQL analysis 503.

Claims (1)

1. An intelligent traffic monitoring analysis system based on big data technology, comprising: the system is characterized in that after entering a platform through a user input website (101), the login module inputs an account number and a password (102) and enters the verification module (2); the back end acquires the account password (201) and enters information and database account password information into a database for verification (202), if the verification is successful (203), the account password is returned to be input (102), the user inputs the account password again, and if the verification is successful (203), the platform (204) is successfully entered; the road condition overview module (3) is composed of data processing (301), data partition storage (302) and data HQL analysis (303), after the road condition data processing is carried out by the data processing (301), the data partition storage (302) is used for carrying out data storage, partition tables are built when Hive is used for carrying out data storage, optimization is carried out simultaneously, parquet format is used for storage, the compression format of the data is snappy, a large amount of data can be rapidly processed, the data HQL analysis (303) is used for carrying out data analysis, the data stored in Hive is analyzed and calculated through HSQL sentences, and when the HQL sentences cannot meet the requirement of the data to be processed, the data is processed by using a custom UDF function; the congestion prediction module (4) consists of data acquisition (401), model prediction (402) and data importing MySQL (403), the data acquisition (401) acquires road congestion data, the model prediction (402) performs data analysis, a lightGBM model is innovatively used, and the model is an efficient gradient lifting decision tree framework, and has the following application process in traffic flow prediction: (1) Preparing a data set, namely preparing the data set required by traffic flow prediction, wherein the data set comprises influence factors of factors such as historical traffic flow, weather conditions, holidays and the like on the traffic flow, preprocessing the data set, including data cleaning, missing value filling and characteristic engineering, and receiving the real-time flow number of a kafka message queue by using spark; (2) Model training, namely training a lightGBM model, wherein the process comprises the steps of selecting characteristics, setting parameters, training the model and verifying, wherein parameter adjustment can be divided into two parts, the first part is to determine basic parameters of a tree model, the second part is to adjust the model training parameters, the basic parameters of the tree model comprise the number of leaf nodes and the learning rate, and the optimal parameter combination is found by adopting a grid searching and Bayesian optimizing method; (3) After model evaluation and model training are completed, the performance of the model is evaluated by using a cross-validation method, the prediction accuracy of the model is validated to be more than 90%, and data is imported into a MySQL (403) and is imported into a platform (204); the safety management module (5) consists of data preprocessing (501), data warehouse design (502) and data HQL analysis (503), the data preprocessing (501) preprocesses safety data, the data warehouse design (502) stores traffic data into a data warehouse, and data layering is creatively adopted in the storage process: (1) A data operation layer (ODS), wherein the data is consistent with the original data downloaded by the official network, the history data stored in the Hive is read-only, and a front-end user and a background system are provided for inquiring the data; (2) The data detail layer (DWD) cleans, standardizes and dimension degenerates the data of the ODS layer, and splits the original data table into a user information table, a vehicle data table, a violation data table and other smaller tables for preparing the next analysis operation; (3) A data summarizing layer (DWS), wherein the data of the data summarizing layer are calculated and summarized according to the analysis subject, the personal information of the user and the corresponding vehicle information data are required to be inquired, the two tables are combined, and a data aggregation is focused on a multi-bin model with better complex inquiry and processing performance; (4) The data application layer (ADS), which is also called as data mart, stores data analysis results, can be used for visual display of data, can also facilitate users to add, delete and examine specific data, lighten the burden of a data warehouse, and analyze and process safety data by the data HQL analysis (503).
CN202310617204.9A 2023-05-29 2023-05-29 Intelligent traffic monitoring analysis system based on big data technology Pending CN116665441A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310617204.9A CN116665441A (en) 2023-05-29 2023-05-29 Intelligent traffic monitoring analysis system based on big data technology

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310617204.9A CN116665441A (en) 2023-05-29 2023-05-29 Intelligent traffic monitoring analysis system based on big data technology

Publications (1)

Publication Number Publication Date
CN116665441A true CN116665441A (en) 2023-08-29

Family

ID=87714706

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310617204.9A Pending CN116665441A (en) 2023-05-29 2023-05-29 Intelligent traffic monitoring analysis system based on big data technology

Country Status (1)

Country Link
CN (1) CN116665441A (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117238145A (en) * 2023-11-14 2023-12-15 山东纵云信息技术有限公司 Intelligent traffic management method and system based on big data
CN117912255A (en) * 2024-03-19 2024-04-19 河北鹏鹄信息科技有限公司 Real-time intelligent driving global data acquisition highway monitoring system and monitoring method

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117238145A (en) * 2023-11-14 2023-12-15 山东纵云信息技术有限公司 Intelligent traffic management method and system based on big data
CN117912255A (en) * 2024-03-19 2024-04-19 河北鹏鹄信息科技有限公司 Real-time intelligent driving global data acquisition highway monitoring system and monitoring method
CN117912255B (en) * 2024-03-19 2024-05-10 河北鹏鹄信息科技有限公司 Real-time intelligent driving global data acquisition highway monitoring system and monitoring method

Similar Documents

Publication Publication Date Title
CN116665441A (en) Intelligent traffic monitoring analysis system based on big data technology
Mavi et al. Bus rapid transit (BRT): A simulation and multi criteria decision making (MCDM) approach
Ferreira et al. The impact of driving styles on fuel consumption: A data-warehouse-and-data-mining-based discovery process
CN110705747A (en) Intelligent public transport cloud brain system based on big data
CN101261722A (en) Electronic police background intelligent management and automatic implementation system
CN111914687B (en) Method for actively identifying accidents based on Internet of vehicles
CN109979220A (en) A kind of high-speed service area intelligence system for automobile trip
CN105260832A (en) Performance evaluation method for taxi drivers based on order data
CN109670673A (en) Strip mine production management and control system
CN104331502A (en) Identifying method for courier data for courier surrounding crowd marketing
CN115018148A (en) Urban rail transit network passenger flow distribution prediction method and system based on digital twin model
CN115662113A (en) Signalized intersection people-vehicle game conflict risk assessment and early warning method
CN114841843A (en) Method and system for analyzing suspicious green traffic
CN106127316A (en) Automated vehicle maintenance prediction system
Sun et al. Research on safe driving behavior of transportation vehicles based on vehicle network data mining
CN111696347B (en) Method and device for automatically analyzing traffic incident information
CN113379318A (en) Method and device for evaluating operation service quality of public transport system and computer equipment
CN109409563A (en) A kind of analysis method, system and the storage medium of the real-time number of bus operation vehicle
CN111582191B (en) Pouring amount estimation method in concrete pouring based on artificial intelligence video analysis
CN116128197A (en) Intelligent airport management system and method
CN201030869Y (en) Railway engineering failure management information system
CN108830509B (en) Method for dynamically adjusting capacity scale of touring taxi
CN111914138A (en) Working condition analysis method, device and system based on Internet of vehicles big data
CN110009117A (en) Mechanism model generation method and device based on train status data
Cohen et al. Assessing the impact of speed limit changes on urban motorways: A simulation study in Lille, France

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination