CN112687097A - Highway highway section level data center platform system - Google Patents

Abstract

One or more embodiments of the present disclosure provide a highway section-level data middling station system, which provides a multi-mode processing computing framework for massive complex data in the field of a road network based on the standardization of elastic multi-source heterogeneous data resources, so as to implement quick identification, accurate fusion, and efficient processing of multi-dimensional data heterogeneous data, provide analysis, prediction, and decision support services for the highway industry by combining artificial intelligence and machine learning technologies, provide intelligent and powerful solutions for management departments to early discovery, early prevention, and early solution of road section traffic conditions and potential dangerous events, share and fuse a data model and a business model based on a micro-service architecture, develop services, support secondary development and access and expansion of third-party partners, and form a business data ecosystem with a data middling station as a core.

Description

Highway highway section level data center platform system

Technical Field

One or more embodiments of the present specification relate to the technical field of traffic data processing, and in particular, to a highway section level data middlebox system.

Background

Through the development and construction of the expressway in China in nearly 20 years, the mileage of the vehicle passing through the expressway reaches 14 kilometers by 2018. At present, daily operation management of highways of various provinces in China is relatively extensive, and various management systems generate rapid, large-amount, multi-source and multi-source heterogeneous data which are not fully mined and utilized. The application of big data analysis technology is only limited to the solution exploration of the hot problem in a specific subdivision field;

the ministry of transportation at present releases the outline of digital traffic development planning in 2019, and proposes that data are key elements and core drivers. Novel information technologies including cloud computing, artificial intelligence, data mining, computer vision, deep learning, and the like are being widely used. The application of new technologies in the traffic industries such as vehicle-to-road coordination, edge calculation and the like is based on efficient management and use of massive data, traffic resources are integrated, cross-domain resource sharing and data opening are promoted, the information construction information island in the field of the current expressway is pervasive, the management efficiency is low, and an integrated, humanized and convenient efficient operation and service ecosystem needs to be constructed.

Disclosure of Invention

In view of this, one or more embodiments of the present disclosure provide a highway section level data middlebox system to solve the problem of low management efficiency due to the fact that information-based construction information islands in the highway field are distributed.

In view of the above, one or more embodiments of the present specification provide an expressway section level data center system, including:

the system comprises an acquisition module and a multi-source data standardization interface, wherein the acquisition module is used for carrying out unification and standardization acquisition on highway data information through the multi-source data standardization interface;

the storage module is used for carrying out distributed storage on the highway data with various different structures acquired by the acquisition module;

the data management module is used for performing data management on the data acquired by the acquisition module and the data stored by the storage module, wherein the data management comprises but is not limited to data resource catalog management, data quality management and data security management;

the data analysis module is used for matching a corresponding AI algorithm model according to the service requirements and in combination with the service scene, and performing data screening and feature extraction on the data stored by the storage module to obtain an analysis result;

the API service module is used for providing an interface for a third-party platform, managing registration, release and scheduling of various scene services, and monitoring and managing all running services;

the portal service module is integrated with an application system, data resources and network resources and is used for providing a uniform access interface for data;

and the operation and maintenance management module is used for monitoring the whole system and the performance, acquiring the comprehensive performance index of the system and providing detailed performance monitoring for various bottom layer resources used by the system.

Preferably, the data source comprises basic data, external data and operation data, the basic data is owned data of highway section companies, the external data provides user macroscopic data information support for intelligent highway management and control, the operation data is used for establishing an intelligent traffic management and control model algorithm, a road section traffic early warning and management and control strategy is formed, and road section traffic state and safety risk assessment management is achieved.

Preferably, the multi-source data standardized interface adopts a Web Service technology in principle, the SOAP transmission protocol is utilized, the data exchange interface specification adopts XML/json format description, and binary data is encoded by adopting a Base64 mode. The data specification includes unified classification coding, unified data acquisition procedures, unified data quality standards, unified data formats, unified metadata standards, and the like.

Preferably, the storage module solves the problem of mass data storage, and the key technologies of full-distributed architecture, data block granularity segmentation, online capacity expansion and reduction, copy backup, common PC hardware applicability and the like support safe online storage of data of scale above PB level, so that safe, low-cost and arbitrarily expandable large data storage becomes possible. The storage module is also used for storing structured, semi-structured and unstructured data. The storage module is further used for storing the data in each data area according to data topics, and the data topics include but are not limited to project topics and tag library topics.

Preferably, the data resource catalog management comprises catalog registration, catalog updating, metadata synchronization, main data synchronization, catalog inquiry, data archiving and recovery, the data quality management comprises data quality inspection rules, data problem display and task management, link guarantee, environment and data quality monitoring and periodic report are provided for a third-party platform, the data security management comprises database state monitoring, database auditing, database risk scanning and access control, the problems of data desensitization, traceability and auditing are solved through hierarchical construction and hierarchical protection, and desensitization and deformation are carried out on sensitive information.

Preferably, the data analysis module is further configured to, after screening the data stored in the storage module, pre-process the screened data, where the pre-process includes: and performing set calibration on the screened data, images and images, and converting the data format.

Preferably, the feature extraction comprises: calculating the gain value of each screened data, sequencing the gain values of the screened data from large to small, outputting the characteristic item with the maximum gain value to a file, vectorizing the original document and the document to be classified, calculating the cosine of an included angle between the original document and the document to be classified, selecting the text most similar to the new text in the training text set, and forming characteristic level fusion.

Preferably, the operation and maintenance management module comprises monitoring of a cluster network, a Hadoop system, resource operation, Web service, database monitoring, hardware equipment and system load, early warning and alarm, and log statistical analysis.

As can be seen from the above, the highway section level data middling station system provided in one or more embodiments of the present specification implements a highway operation management service data deep fusion component technology, which is beneficial to a service system to deposit on a middling station in a modular manner, and opens in a fusion component mode, so that the cost and threshold for building and developing a newly added service module are reduced, the development time is saved, the accumulation of enterprise experience capability is implemented, the system is better adapted to a multi-pass market environment, and the strain capability and the competitiveness are improved.

The method for realizing the design of the highway section level data middlebox is beneficial to forming a standardized data command scheduling service central center platform integrating data resource management, data sharing exchange and data standardized open service by researching the highway cross-field, cross-department and cross-platform data middlebox technology, constructing a data comprehensive command scheduling and analysis service technology base stone, promoting the deep application of a big data technology in the highway daily operation management decision, and effectively improving the highway daily operation management decision level.

Drawings

In order to more clearly illustrate one or more embodiments or prior art solutions of the present specification, the drawings that are needed in the description of the embodiments or prior art will be briefly described below, and it is obvious that the drawings in the following description are only one or more embodiments of the present specification, and that other drawings may be obtained by those skilled in the art without inventive effort from these drawings.

FIG. 1 is a design diagram of a data center system architecture according to one or more embodiments of the present disclosure;

FIG. 2 is a schematic diagram of a data exchange service of an acquisition module, a storage module, and a data exchange module according to one or more embodiments of the present disclosure;

FIG. 3 is a flow diagram of data resource management by a data management module in accordance with one or more embodiments of the present disclosure;

FIG. 4 is a diagram illustrating an example application of an algorithm model of a data analysis module in accordance with one or more embodiments of the present disclosure;

FIG. 5 is an application development process management diagram of an API service module in accordance with one or more embodiments of the present disclosure;

FIG. 6 is a schematic diagram of user services of a portal service module in accordance with one or more embodiments of the present disclosure;

fig. 7 is a system security management and control diagram of the operation and maintenance management module according to one or more embodiments of the present disclosure.

Detailed Description

To make the objects, technical solutions and advantages of the present disclosure more apparent, the present disclosure is further described in detail below with reference to specific embodiments.

It is to be noted that unless otherwise defined, technical or scientific terms used in one or more embodiments of the present specification should have the ordinary meaning as understood by those of ordinary skill in the art to which this disclosure belongs. The use of "first," "second," and similar terms in one or more embodiments of the specification is not intended to indicate any order, quantity, or importance, but rather is used to distinguish one element from another. The word "comprising" or "comprises", and the like, means that the element or item listed before the word covers the element or item listed after the word and its equivalents, but does not exclude other elements or items. The terms "connected" or "coupled" and the like are not restricted to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "upper", "lower", "left", "right", and the like are used merely to indicate relative positional relationships, and when the absolute position of the object being described is changed, the relative positional relationships may also be changed accordingly.

For the current problems and characteristics of the information-based construction information island in the field of the current expressway, low management efficiency and the like, the invention aims to solve the following technical problems:

1. how to build a standardized, streamlined, normalized data system?

The data is not what is seen, but what is used. The big and unified data center only completes the first step of data access integration, unified aggregation and storage, and a standardized, streamlined and normalized data system, a new framework of a data world and global cognition are required to be constructed to fully exert the application value of massive multi-source heterogeneous data in the data center and provide services.

Elements of a data system comprise a whole body, a technical level, a service level and a management system level; the construction of the data system is a deconstruction, reconstruction, innovation, optimization and integration construction process, and the data system, the service system and the organization system are coordinated and managed to establish a data command and scheduling platform; establishing standards, rules and processes, and managing data logic, service logic and how to realize data minimum granularity, standardized association and responsibility system; in order to manage data, directories, tags, clear applications, and theme services are also managed. Through the new data system, a long-acting mechanism integrating collection, aggregation, fusion, treatment, development and maintenance can be really formed. The system is absolutely not a life body but a simple assembly and association of all systems. A one-stop data resource management and service platform is constructed, data quality is comprehensively controlled, data resource standardized management capacity is improved, the phenomenon of data chimney and information island caused by non-sharing of traditional data information is improved, the maximum value of data is fundamentally exerted, the data island is broken, and large industrial global data fusion is completed.

2. How to support industry cross-department, cross-domain, cross-regional data resource sharing exchange services?

At present, with the needs of cross-department, cross-field and cross-region fine management, more and more units need to perform business interaction cooperation to achieve the goal of improving the management level and further achieving fine management. The basis of service collaboration is that information systems can be interconnected and intercommunicated, information resource sharing and exchanging are normalized step by step, and information resources are synchronized, so that the problems of asymmetric information resources and unbalanced information resource distribution are solved, the service deep collaboration can be performed, and the service fusion can be achieved in the future.

Then, in order to implement the cross-department, cross-domain data resource sharing exchange service, the problem of 4W1H is to be solved first: i.e. What the data is (What), Where the data is (Where), Who the data needs (Who), What value the data has (Why), How the data shares the exchange (How).

However, the existing data sharing and exchanging method is still in the situation of function fragmentation, a novel and passive response, scientific integration and integration are not performed between system application modules, a standardized data resource sharing and exchanging system is not established yet, and data sharing and exchanging is far short of a closed-loop intelligent sharing application and management mode. Therefore, the requirements of the multiple departments on standardization, high efficiency, convenience, high quality service, fine management and the like of the information resource sharing and exchanging process which are increasingly growing cannot be met, and the increasing information resource sharing and exchanging requirements of the multiple departments cannot be efficiently supported. The research is necessary to construct a sharing exchange service system with unified flow, reliable standard, safety and high efficiency, and the needs of sharing exchange service of data resources across departments, fields and regions on the highway are met from the aspect of functional support such as transaction flow, resource standard, authority control, quality monitoring and the like.

3. How to improve the comprehensive monitoring, coordination and analysis service level of the highway operation management department?

At present, related service management systems are built in various fields of highway charging, maintenance, daily operation management, safety emergency and the like to different degrees, but in the face of various travel requirements of the masses, new-state cross-department cooperation and multi-department emergency disposal requirements of safety production accidents, the defects of insufficient comprehensive monitoring and coordination capacity of the existing industry, lack of integration of data resources of the existing service lines and lack of shared access to external data resources are exposed. Industry management has long been the mode of "problem-solving", but it is the ultimate goal of industry management to avoid the problem, which requires that industry management have proactive and preventive capabilities. But the current industry comprehensive analysis capability is insufficient, the industry trend can not be scientifically researched and judged, and the industry management decision is difficult to assist. In addition, the travel service of the industry is still dispersed in each business line at present, an integrated travel service system cannot be provided, and the travel requirement of diversity of the masses cannot be met.

Therefore, information intercommunication and service linkage between different departments, including information intercommunication and coordination linkage inside and outside the highway system, need to be established in different industries and different areas to establish an information sharing and service coordination linkage mechanism, so that information is smooth between all levels of administrative departments and all service departments, linkage information is reported between the traffic departments and other government functional departments in time, and the comprehensive monitoring, coordination and analysis service level of the highway operation management departments is greatly improved.

4. How to promote intensive construction of industry information foundation support environment and reduce repeated investment?

At present, independent construction is mainly used for information construction of service parts in each subdivision management field of a highway, and each unit respectively constructs an application system and an application supporting environment according to the service requirements of the department, so that each service system becomes an information isolated island, and certain difficulty is caused for exchange sharing and overall application of data resources. The key for solving the problems is to realize the convergence, integration and application of industrial data resources. A one-stop standardized data command and dispatch service center platform is constructed, and after the platform is constructed, high function integration, information integration and resource sharing can effectively reduce the repeated construction investment of each management department system and the related system maintenance cost, thereby reducing the corresponding capital expenditure.

Therefore, the current situation of highway section companies is researched, the service flow and the current situation of data are analyzed, the requirements of cross-field, cross-department and cross-platform service collaboration, data exchange sharing and decision management existing in the section companies are found out, meanwhile, the similar excellent platform in China is researched, the previous projects are summarized and reviewed, and the collection and query of relevant data of the projects are carried out, so that the experience reference is carried out.

Secondly, the system framework research of the highway operation management middle platform system is carried out, the data middle platform is systematically researched and designed from the aspects of business architecture, functional architecture, technical architecture and the like, a system framework which is fit with the highway operation management characteristics and the data processing application requirements is formed, and six unified business operation modes are realized, namely:

(1) unified service resource allocation: unified configuration of multi-tenant modes of resource application, allocation, supervision and recovery;

(2) unifying data management flow: standardization of a full life cycle management process from access, storage, cleaning, processing to exchange service;

(3) unified data directory management: unified management of data access, integration, cleaning and classified cataloging based on a multilevel directory;

(4) unified data exchange service: unifying the on-demand distribution and exchange service of multi-source heterogeneous data resources;

(5) unifying application development specifications: unified application design development, test, deployment operation and maintenance management specifications;

(6) unified system security management and control: unified authentication, account number, authorization and audit management, and comprehensive guarantee of system safety.

Then, developing a highway section level data middling platform, including acquisition and storage, data management, AI models, API services, portal services and operation and maintenance management, and compiling to form a highway big data middling platform technical standard specification; the functional planning strategy is summarized as "five ones", namely:

(1) the One Pipe provides the capability of data extraction and integration and solves the problems of data extraction and treatment;

(2) one Data provides the capability of Data asset integration and management, and solves the problem that Data is precipitated as assets;

(3) the One AI provides the intelligent prejudging capability, constructs a traffic brain and solves the problem of transition from informatization to intellectualization;

(4) one Service, namely establishing open embodiment of capacity, implanting data into a Service, solving the problems of Service datamation and data value;

(5) and (3) establishing a unified maintenance and operation support platform, refining and standardizing operation and maintenance, and solving the problems of system expansion and high availability.

The embodiment of the specification discloses a highway section level data middling station system, which comprises an acquisition module, a multi-source data standardization interface, a storage module, a data exchange module, a data management module, a data analysis module, an API service module, a portal service module and an operation and maintenance management module, wherein the acquisition module is used for unifying and standardizing highway data through the multi-source data standardization interface, the storage module is used for storing the highway data acquired by the acquisition module in a classified mode according to data sources, for example, the data acquisition relates to unified interfaces and data preparation of off-line data and (quasi) real-time data, the real-time data synchronization function is realized, data increment or full-scale synchronization is supported, and a system interaction mode supports synchronous and asynchronous modes. The method is oriented to storage, processing and migration of the quasi-PB-level mass heterogeneous highway big data, realizes calculation or storage of real-time received distributed message queue flow processing, can store unstructured data after structured processing, supports accumulation, extraction, cleaning and the like of historical data, and realizes information distributed storage according to data service requirements.

The traffic big data is stored in a partitioned mode according to a data source and is divided into a basic data area, an external data area and an operation data area, data in each data area are stored according to a data theme, and the data theme supports user customization.

(1) Basic data: the data from the system owned by the road company and generated in the operation process belong to the owned data of the highway section company. The method comprises the following steps: the system comprises basic road section GIS data, cross-dispatching station data, bridge and tunnel basic data, historical transaction detail data, historical accident and flow data information, video monitoring data, service area data, road planning data, road maintenance data and the like.

(2) External data: and providing user macroscopic data information support for intelligent management and control of the highway. The method comprises the following steps: weather data, traffic police system data, network information data, user usage review data, platform data such as vehicle networking, other collaborative or purchasing data, and the like.

(3) And (3) operating data: and establishing an intelligent traffic control model algorithm to form road section traffic early warning and control strategies and realize road section traffic state and safety risk assessment management. The method comprises the following steps: short-term road condition prediction data, traffic incident study and judgment feature data, key vehicle identification track data, income prediction data, section flow short-term prediction data, risk identification data and the like.

Data topics include project topics, tag library topics, other topics, and the like.

The data center system provides a uniform data access mode, and multi-source acquisition and uniform storage processing are realized. The data type should support both structured and unstructured data acquisition. The collection mode simultaneously supports batch collection of periodic interface files and collects data with high generation frequency and strong real-time property in a data stream mode.

The data exchange is used as an important component of the integrated platform and mainly realizes the data exchange among the platforms and the businesses. The system combines practical application to formulate a data interface technical specification according to national/industrial standards. The system is large in industrial data size and multiple in data types, so that the establishment of the interface standard specification has great significance. The technical specifications mainly comprise access service specifications, access service input parameter specifications, access service return value specifications, data arrival notification data formats, exchange log specifications and the like. In principle, a Web Service technology is adopted, a SOAP transmission protocol is utilized, the data exchange interface specification is described by adopting an XML/json format, and binary data is encoded by adopting a Base64 mode. The data specification includes unified classification coding, unified data acquisition procedures, unified data quality standards, unified data formats, unified metadata standards, and the like.

The requirements of the multi-source data standardized interface meet the following contents:

(1) the data interaction is supported at an enterprise level, and safe and reliable interaction is provided on the basis of high concurrency and large capacity of the system.

(2) And a perfect information safety mechanism is provided to realize the comprehensive protection of information and ensure the normal operation of the system. The situations of massive access and massive occupation of resources are prevented, and the robustness of the system is ensured.

(3) An effective, systematic, monitorable mechanism is provided so that the operational condition of the interface can be monitored for timely error detection and troubleshooting.

(4) On the premise of fully utilizing system resources, smooth transplantation and expansion of the system are realized, and dynamic expansion of the system resources is provided when the system is concurrently increased so as to ensure the stability of the system.

(5) When the capacity expansion and the new service expansion are carried out, a quick, convenient and accurate implementation mode can be provided.

(6) The interface technology implementation remains neutral.

(7) Provides the function of automatically marking the source of input and output in information exchange.

The interface communication is based on the mainstream communication protocol and satisfies the following:

(1) the data transmission is controllable and provides a data retransmission function.

(2) The data transmission has reliability, data can not be lost, and sufficient data verification is performed.

(3) The big data transmission has the function of breakpoint continuous transmission.

The interface mode management should satisfy the following:

(1) the information exchange mode conforms to the XML data exchange standard.

(2) The interoperation service interface conforms to the WebServices standard.

(3) The system interaction mode supports both synchronous and asynchronous modes.

(4) Interactive data supports various data types.

The data interface model consists of a data structure, a data set and an accessory set:

(1) the data structure is used to describe the structural information of the interface, and is an optional element.

(2) A dataset is used to encapsulate structured data and is an optional element.

(3) The attachment set is used for expressing unstructured data and is an optional element.

(4) The data set and the attachment set may co-exist or appear separately.

The storage module is a storage center of global data provided by the data center system, and carries out mixed matching on various data storage systems according to different data types, so that efficient, stable and safe data storage is realized; establishing an analytical data warehouse, and performing data mining on massive complex data; establishing a relational database to support service interaction circulation; the complex format of the data is processed by using a data lake technology, and the problem of data isolated island of an enterprise is solved.

And the video networking platform is used for converging all the cameras on the road network, acquiring data and controlling equipment, and the equipment access platform is used for accessing all the detectors and the issuing equipment of the road network, acquiring data and controlling the data.

For data of an existing platform in the industry and data acquired in other ways, the basic data platform can connect all related business systems with the exchange platform through adapters to acquire required data of various structures.

The introduction of the distributed system solves the difficult problem of mass data storage, and the key technologies of full distributed architecture, data block granularity segmentation, online capacity expansion and reduction, copy backup, common PC hardware applicability and the like support safe online storage of data of scale above PB level, so that safe, low-cost and arbitrarily expandable large data storage becomes possible. Structured, semi-structured, unstructured data may be stored.

(1) Structured data store:

a structured data storage area is established on a data resource center, traditional relational data are stored and managed, a basic library, a business library and a subject library are established, business data such as production operation application, traffic safety application, public service application, aid decision application and the like are supported, and data support is provided for basic data products, fixed report business and business intelligent application. Time series data storage area

And customizing and developing a time sequence data storage area to store and manage mass time sequence monitoring data and provide data service capabilities such as efficient query processing and the like. The time sequence data has the characteristics of large data volume, high access average rate, remarkable data time sequence characteristics, uninterrupted data access and the like, and the access mode of the time sequence data also has remarkable time sequence-based characteristics.

(a) Time series data storage

And customizing and developing a time sequence data storage area to store and manage mass time sequence monitoring data. Time series data metadata management

And unified management on the time sequence data metadata is realized. And the user is allowed to model the metadata according to the actual use scene and the data model, so that the data model is unified, and the data access and use are simplified. And multi-version maintenance of data is supported, and when the model is changed, the data before and after the change can be effectively stored on the system.

(b) Time-series data time range query

And the special query function of the time series data is supported, and comprises a query of a given time range on an original data value, a query of an aggregation operation of data in the given time range and the like.

The original data value query refers to selecting a domain, starting time and ending time, and querying actual values of some indexes in the time range; and the aggregation operation query refers to a selected domain, and a start time and an end time, and aggregation (such as sum, count, max, min) is performed on the actual values of some indexes in the time range.

(c) Time series data fuzzy point query

And the method supports that the index value near the time is inquired at the given time point on the time series data. Including forward, backward, forward and backward, queries of the nearest data points, where the number of query data points may be a single point or multiple point query.

(d) Multi-dimensional filtering query of time series data

The multidimensional filtering query function supporting time sequence data comprises the steps of appointing value ranges with different attributes to carry out joint filtering query, carrying out aggregation query on data meeting filtering conditions, carrying out scaling on data values of the time sequence, carrying out alignment query among the time sequences and the like.

The joint filtering query refers to selecting a point domain, setting a starting time and an ending time, setting the value taking conditions of certain indexes (the value taking conditions of the indexes can be any combination of or and), and then performing value query on certain indexes meeting the conditions;

the aggregation query is to perform aggregation (such as sum and average) operation on the index data meeting the filtering condition.

Scaling of time series data values refers to converting the values that meet the filtering criteria (e.g., multiplying all results by 10).

The alignment query refers to performing alignment operation on query results according to a given time sequence, so that data values of all indexes fall on the same time.

(e) Reverse lookup of time series data instances

And a time sequence data instance back-check function is supported, namely, an object or asset instance corresponding to the data meeting the specified data condition is found according to the specified data condition. Such as: given some indexes and the numerical range of the indexes, the indexes can be arbitrarily combined through or and, the starting time and the ending time are set, and all domains meeting the conditions are inquired.

(2) Semi-structured data storage:

massive semi-structural data and structural data such as meteorological, internet public sentiment, Web and social media and the like, and large-scale data sets such as intermediate process and result data generated in the big data analysis process have multiple data types, and a semi-structural database is built, for example, HBase stores the data. The HBase has the advantages of easiness in expansion, high cost performance, low operation and maintenance cost and the like, and meanwhile, a deployment cluster machine of the HBase can use a common SATA disk to support massive semi-structural data and structured data. The advantage of the NoSQL access scheme of HBase for semi-structured, structured data is as follows:

(a) the columns of HBase can be dynamically increased, so that the method is well suitable for application scenes of semi-structured data, namely uncertain storage modes;

(b) when the column is empty, data are not stored, and the waste of storage space is avoided;

(c) multiple versions of data may be stored;

(d) automatic expansion is realized through HRegion dynamic Split and Merge, and high availability is realized by using a plurality of data backup functions provided by an HDFS;

(e) cluster expandability, wherein the HBase stores data based on the HDFS, and when the data volume is increased to the extent that the current cluster cannot bear the data volume, the cluster expandability based on the HDFS can be realized, and horizontal expandability can be conveniently realized by adding machines;

(f) HBase adopts a strong consistency model, and after one write return, all reads are ensured to read the same data;

(g) the HRegionServer and the DataNode are operated on the same server to realize the localization of data, improve the read-write performance and reduce the network pressure;

(h) the downtime of the built-in HRegionServer is automatically recovered. Adopting WAL to Replay data which is not persisted to HDFS;

(i) the method can be seamlessly integrated with Hadoop/MapReduce, and the management of NoSQL metadata is realized through an encapsulation interface. And the semi-structure data access service is provided to the outside by packaging the corresponding interface.

The data in the big data support system comprises unstructured and semi-structured data with mass scale (PB level) besides traditional structured data, such as pictures, surveillance videos, standard specifications and the like. Oriented to massive unstructured and semi-structured data storage, the traditional centralized and array storage mode cannot meet the storage requirement of massive data, and the problems of poor capacity expansion, poor reliability and high availability and the like exist. The introduction of the distributed file system solves the problem of mass data storage, supports safe online storage of scale data above PB level, and makes safe, low-cost and arbitrarily expandable mass data storage possible.

Distributed file systems refer to systems that are based on a client/server model, where the physical resources managed by the file system are not necessarily directly connected to a local node, but rather are connected to the node through a computer network. The distributed file system is characterized in that file data are stored on scattered low-cost storage media, a consistent file access interface is provided for the outside, and the distributed file system has good fault tolerance.

The data exchange module is used for sending the highway data information stored by the storage module to a data caller and acquiring the highway data information from a third-party platform, for example, batch exchange of data is carried out in batches through an ELT platform, the batch exchange comprises receiving and sending preprocessed data, the data is sent to an open test cluster and is issued to business applications, service system service calling is realized through Web service calling, and real-time monitoring of a business system is carried out in a message queue mode.

The data center system provides uniform big data computing capacity, provides a data computing framework model matched with different big data processing scenes, and mainly comprises batch offline computing (MapReduce and the like), memory computing (Spark, Ignite and the like), online Streaming computing (Storm, Samza, Spark Streaming and the like), and prediction learning computing (Mahout, Spark MLib, Caffe, Keras, TensorFlow and the like).

(1) Real-time property

In the fields of road network running states, traffic flow and the like, the collected key index data is generally required to be calculated in time, and real-time data calculation is carried out according to a preset data rule by receiving real-time data sent by a real-time data interface.

(2) Parallelism

The data collected by the highway domain is generally stored in a TB level or a PB level. For data of such magnitude, almost any non-parallelized analysis engine cannot calculate and analyze the result in a valid time. The parallelization analysis engine effectively parallelizes the data to be analyzed to execute calculation and analysis, so that a calculation analysis result is obtained within feasible time.

The data management module is used for performing data management on the data acquired by the acquisition module and the data stored by the storage module, the data management includes but is not limited to data directory management, data quality management and data security management,

the data center station provides data asset management, realizes acquisition, management and control, delivery and the like of data asset values, formulates data specifications, specifies data apertures, common terms, reference data, coding modes and the like in detail, and realizes standardization of data management; establishing a data model, and performing unified planning and control on a logic model, a physical model, a database table, fields, views and the like of a core in the system; unified metadata management, including business metadata, technical metadata, process metadata, data management system metadata and the like; establishing a perfect data quality management system, including technical audit, quality measurement, effect evaluation and the like, so as to improve the quality level of data; and establishing a data security management mechanism, formulating a data security strategy, and providing security management methods such as data authentication, authorization and audit.

The data resource catalog management comprises functions of catalog registration, catalog updating, metadata synchronization, main data synchronization, catalog inquiry, data archiving and recovery and the like, and the collected and stored data supports the cataloging according to data sources, business subjects, life cycles, service modes and the like.

The data quality management provides link guarantee, environment and data quality monitoring and periodic reporting for the platform, and comprises functions of data quality inspection rules, data problem display and task management, so that enough high-quality data can be guaranteed, and a foundation is laid for data fusion and analysis.

The data security management solves the problems of data desensitization, tracing and auditing through layered construction and hierarchical protection, desensitizes and deforms sensitive information, improves the working efficiency of data management personnel, avoids data leakage risks, has various engines such as database state monitoring, database auditing, database risk scanning and access control, and provides perfect protection for the security of client information assets and the protection of sensitive information.

The data analysis module is used for screening and extracting the characteristics of the data stored in the storage module according to the service requirements by combining with the service scene and matching with the corresponding data fusion AI algorithm to obtain an analysis result, and is also used for preprocessing the screened data after screening the data stored in the storage module, wherein the preprocessing comprises the following steps: and performing set calibration on the screened data, images and images, and converting the data format.

For example, the data analysis module can adopt an AI model system, and combine with the existing service application scenarios such as event detection, road condition patrol, flow simulation, flow prediction, bridge anomaly detection, health assessment, etc., to realize the deep utilization of data resources, and achieve the purpose of driving service operation and completing knowledge precipitation of enterprises by mining effective information, establishing a knowledge map, and forming data assets.

The data fusion processing solves the problems that the user data is split and the user appearance cannot be comprehensively outlined, the data split causes the user to be recognized more comprehensively, wrong decisions can be made, different source service data are fused, the data complementarity and integrity are enhanced, the data connotation value is effectively improved, the dimensionality modeling mainly realizes the integration of crossing data, and the integration form is summary, association, analysis and the like.

The data fusion utilizes a logic algorithm, a computer technology and the like to control and integrate multi-source heterogeneous data (structured data and semi-unstructured data), so that a more accurate and comprehensive information result is obtained. The most important reason for data fusion is the fracture of user data, which cannot fully outline the user's entire appearance, and the fracture of data causes the user to be more cognized and possibly make wrong decisions. The data fusion utilizes relevant means to integrate all information obtained by investigation and analysis, and carries out unified evaluation on the information to finally obtain unified information, aiming at integrating various different data information, absorbing the characteristics of different data sources and then extracting unified information from the information, wherein the unified information is better and richer than single data.

When the multi-source data fusion technology is applied to high-speed big data, the optimal selection of the data can be realized, and the result can be obtained by preprocessing the data, then performing data fusion and comprehensive analysis. The method is a characteristic of the multi-source data fusion technology and enables the specific application operation process to be realized.

Data selection is the first step of multi-source data fusion, and must ensure the correctness of data selection to optimize a proper data object as much as possible for data fusion, and if the data object is selected incorrectly, the later fusion effect of the multi-source data is directly influenced. After the data is reasonably selected, the data needs to be preprocessed firstly, so that the fusion effect of unprocessed data is prevented from being influenced during multi-source fusion, and the data obtained after fusion cannot meet the business application requirements. The data preprocessing method mainly comprises the following steps: firstly, the selected various data, images and images are geometrically calibrated, and then the data format is converted. Preprocessing is necessary to ensure that all data can be communicated with each other, and data fusion can be realized on the same platform.

The feature selection needs to calculate the gain value of each data according to the file obtained in the previous step, sort the information gain of each data from large to small, and finally output the feature item with the maximum gain value to the file. The original document and the document to be classified need to be vectorized, the original document needs to be normalized after vectorization, and the document to be classified does not need to be normalized after vectorization, so that a vectorized class is provided for the original document and the document to be classified respectively. And reading the feature items from the file, and storing the feature items left after feature selection in the parameter file. After the original documents are vectorized, the contents output to each document are different, and the documents to be classified do not need to be normalized after the vectorization. And then, in the classification step, the cosine of an included angle between the original document and the document vector to be classified is calculated. The larger the final absolute value, the smaller the included angle, the more similar, and the closer the distance. And selecting the text most similar to the new text in the training text set, wherein the document belongs to the most class, and the documents to be classified belong to the most class, so that feature level fusion is formed.

After the data is processed, the application data is used for guiding the high-speed service application from the perspective of data mining. Data mining is a multidisciplinary cross field, and the technology is closely related to disciplines such as statistics, machine learning, neural networks, pattern recognition, knowledge base systems, information retrieval, high-performance computers and visualization. There are many terms that are similar to the meaning of data mining, such as knowledge discovery, knowledge extraction, data/pattern analysis, and the like. Fayyad et al's definition of "knowledge discovery" can be used to interpret the technical meaning of data mining: "identifying an efficient, novel, potentially useful, and ultimately understandable non-trivial process of patterns from a dataset". Where "nontrivial" indicates that the process is not obvious, but rather a process that requires a significant amount of effort to achieve results. From the perspective of high-speed operation and maintenance, data mining is a novel information processing technology, and is mainly characterized in that a large amount of service data in a high-speed operation database are extracted, converted, analyzed and processed, and knowledge for assisting operation decision is extracted from the service data.

In data mining technology, "knowledge" is described as "useful understandable patterns", and the data mining task is to find the "knowledge patterns", and the description of the data mining task is also diversified according to different types of the knowledge patterns. Common data mining tasks include classification analysis, cluster analysis, trend analysis, association analysis, correlation analysis, outlier detection (or outlier detection), mathematical modeling, and the like. The trend analysis is to obtain a corresponding trend line by sorting the data on the basis of mass data for a long time, and estimate the development trend of the monitored data according to the trend line; the correlation analysis is used for analyzing the relation among the monitored data, and the exact relation among the monitored data is difficult to define under the existing technology and conditions for massive data, and the correlation analysis has important significance as long as the correlation relation among the monitored data is known; correlation analysis refers to the inherent relationship between parameters.

The method supports increased business application analysis requirements by continuously precipitating algorithm models such as association rule learning, artificial neural networks, deep learning algorithms and the like, is oriented to business application, excavates information from a large amount of data, completes construction of highway knowledge maps by four processes of information extraction, knowledge representation, knowledge fusion and knowledge reasoning, reveals meaningful relations, trends and patterns among data based on artificial intelligence, pattern recognition and machine learning search algorithms, modeling technologies and learning theories, and plays roles of prediction and decision assistance for business application.

For example, ETC fare evasion inspection is supported by big data and machine learning analysis, ETC vehicle running time screening, ETC vehicle passing frequency screening, vehicle running track tracking, image feature extraction, vehicle searching with a graph, vehicle feature tag library establishment and other feature extraction analysis are carried out, and through algorithm model calculation such as an image recognition algorithm, a traffic event recognition algorithm, an event study and judgment algorithm, a track tracking algorithm and the like, business targets and capability precipitation are continuously optimized, expanded and perfected in a business scene, and further project construction and enabling are assisted.

The API service module is used for providing an interface for a third-party platform, managing registration, release and scheduling of various scene services, and monitoring and managing all running services.

Specifically, the API service module provides service-oriented predefined functions or application program services based on data management and AI models, users do not need to contact original data or original codes, application thresholds are reduced, and the operation is simple and convenient. The data center station can provide reusable public technical components or product services related to the business, such as data analysis, data release, prediction learning calculation and the like, support various rich data display modes, simultaneously directly provide services to the outside by using an SAAS mode, provide fine-grained functional interfaces such as API (application programming interface), message interfaces, file interfaces, service interfaces, SDK (software development kit) and the like, support secondary development and access and expansion of third-party partners, and form a data ecosystem taking the data center station as a core.

The information resource sharing exchange follows a standard Service Oriented Architecture (SOA) mode, and based on an advanced Enterprise Service Bus (ESB) technology, the data sharing exchange service comprising the operations of sharing exchange application and audit, a sharing exchange service portal, a sharing exchange basic support component and the like is provided for interconnection and intercommunication among different application systems and different databases across regions, departments and platforms, and channels and environments of information resource sharing exchange are constructed, the sharing exchange task can be scheduled and managed, the sharing exchange process can be audited, and the problem of sharing exchange abnormity can be traced.

The visual open service is designed, researched, configured, issued and shared based on various plug-ins such as query, layout, panel, template, legend, map, video and development, statistics and query can be carried out through a user-defined query scheme, a data report is generated, a template display area can be issued, application configuration of an analysis legend can be issued, a complete visual page can be issued, and abundant data visual customization service is provided. And providing interfaces for each service system based on an application service architecture, managing registration, release, scheduling and the like of various scene services, and monitoring and managing all running services.

The portal service module is integrated with an application system, data resources and network resources and is used for providing a uniform access interface for data.

The portal service module relates to a functional module of highway big data computing service, a related algorithm and a detailed data structure, and comprises modules and internal details of user login, data display, computing service, statistical analysis, operation configuration, access control, background management and the like, so that the use threshold of a big data platform is reduced, unnecessary time waste during the use of a computing tool is saved, and safety and reliability of related authentication, authorization, encryption, audit and the like are realized. The method is characterized in that a pure personalized information window which can be freely customized by a user is centered on the user, and a single-page entrance is provided for the user, and the functions of single sign-on, business handling, to-do matters, public information, system navigation, information exchange, my warning, information bulletin, information publishing and the like are included.

The portal service module provides a uniform access interface for data stored on the big data support system, and allows a user to provide data access services for the big data analysis application through the services in the big data support system. The user can access the data of the network through the high-speed service module. The system provides a uniform data product access interface on the basis of original data through specific rules or data products formed by calculation and processing, such as business reports, data evaluation, data analysis and thematic analysis.

The unified portal service can provide more reasonable data and application resource distribution, realize unified 'human integration, interface integration, flow integration, business integration, message integration and application integration' by integrating an application system, data resources and network resources under a unified information portal, provide a unified information resource access entrance for owners, travelers and management units, and provide personalized services according to different roles of users.

The operation and maintenance management module is used for monitoring the whole system and the performance, acquiring the comprehensive performance index of the system and providing detailed performance monitoring for various bottom layer resources used by the system. The operation and maintenance management module comprises the functions of monitoring a cluster network, a Hadoop system, resource operation, Web service, database monitoring, hardware equipment and system load, and early warning and alarming and log statistical analysis.

For a complex system such as a data console, not only the overall performance of the system needs to be monitored to obtain the comprehensive performance index of the system, but also detailed performance monitoring needs to be provided for various bottom layer resources used by the system, so as to ensure that the application on the upper layer can continuously call the system service and ensure that the bottom layer data can be accessed to a big data support system. These performance indicators can be used not only to evaluate the current system operating conditions, but also to provide a data source for resource planning and performance alarms.

The standardized operation and maintenance is based on an IT operation and maintenance standardization system, aiming at problem events and operation and maintenance tasks such as portal authority faults, exchange sharing faults, statistical analysis faults, database faults, system software faults, network faults and the like, the operation and maintenance management is carried out in an online mode, the functions of task creation, task list issuing, task registration, task query, event reporting and the like are achieved, and the safe and reliable operation of platform software is guaranteed.

The data center station system disclosed by the embodiment of the specification has the following characteristics:

(1) the method for managing the data resources based on the elastic multilevel directory and the multimode cooperative processing mode of the mass data realize the standardization of multisource heterogeneous data resources of the highway: for massive heterogeneous complex data generated in the field of road networks, including image data, text data, multimedia stream data and the like, a data center station provides a multi-mode processing and computing framework which is tightly combined with data characteristics of highway operation, realizes the purposes of utilizing lower hardware computing resources and realizing the purposes of quickly identifying, accurately fusing and efficiently processing heterogeneous data of multi-dimensional data through collaborative integration and solving the problem of quickly fusing complex data in the field of road network data.

(2) The artificial intelligence and machine learning technology provides analysis, prediction and decision support services for the highway industry: the intelligent algorithm model is applied to scenes such as traffic situation monitoring, video event detection, bridge and tunnel detection, various traffic data display and analysis and real-time prediction are realized, automatic alarm and plan generation are carried out when the data exceed a threshold value, and intelligent and powerful solutions are provided for early discovery, early prevention and early solution of road traffic conditions and potential dangerous events for management departments.

(3) Sharing, fusing and developing services between a data model and a business model based on a micro-service architecture: based on the micro-service architecture, reusable public technical components or product services related to the service, such as data analysis, data release, prediction learning calculation and the like, are provided, various rich data presentation modes are supported, meanwhile, the SAAS mode is used for directly providing the service to the outside, fine-grained functional interfaces such as API (application programming interface), message interface, file interface, service interface, SDK (software development kit) and the like are provided, secondary development and access and expansion of a third-party partner are supported, and a service data ecosystem taking a data center as a core is formed.

The data center system realizes the deep fusion component technology of the highway operation management service data, is beneficial to the service system to deposit on the center in a modularized mode, is open in a fusion component mode sharing mode, reduces the construction and development cost and threshold of newly added service modules, saves the development time, realizes the accumulation of enterprise experience capacity, better adapts to the market environment for multiple times, and improves the strain capacity and the competitiveness.

The method for realizing the design of the highway section level data middlebox is beneficial to forming a standardized data command scheduling service central center platform integrating data resource management, data sharing exchange and data standardized open service by researching the highway cross-field, cross-department and cross-platform data middlebox technology, constructing a data comprehensive command scheduling and analysis service technology base stone, promoting the deep application of a big data technology in the highway daily operation management decision, and effectively improving the highway daily operation management decision level.

Those of ordinary skill in the art will understand that: the discussion of any embodiment above is meant to be exemplary only, and is not intended to intimate that the scope of the disclosure, including the claims, is limited to these examples; within the spirit of the present disclosure, features from the above embodiments or from different embodiments may also be combined, steps may be implemented in any order, and there are many other variations of different aspects of one or more embodiments of the present description as described above, which are not provided in detail for the sake of brevity.

In addition, well-known power/ground connections to Integrated Circuit (IC) chips and other components may or may not be shown in the provided figures, for simplicity of illustration and discussion, and so as not to obscure one or more embodiments of the disclosure. Furthermore, devices may be shown in block diagram form in order to avoid obscuring the understanding of one or more embodiments of the present description, and this also takes into account the fact that specifics with respect to implementation of such block diagram devices are highly dependent upon the platform within which the one or more embodiments of the present description are to be implemented (i.e., specifics should be well within purview of one skilled in the art). Where specific details (e.g., circuits) are set forth in order to describe example embodiments of the disclosure, it should be apparent to one skilled in the art that one or more embodiments of the disclosure can be practiced without, or with variation of, these specific details. Accordingly, the description is to be regarded as illustrative instead of restrictive.

It is intended that the one or more embodiments of the present specification embrace all such alternatives, modifications and variations as fall within the broad scope of the appended claims. Therefore, any omissions, modifications, substitutions, improvements, and the like that may be made without departing from the spirit and principles of one or more embodiments of the present disclosure are intended to be included within the scope of the present disclosure.

Claims (8)

1. A highway section level data middlebox system, characterized by includes:

the system comprises an acquisition module and a multi-source data standardization interface, wherein the acquisition module is used for carrying out unification and standardization acquisition on highway data information through the multi-source data standardization interface;

the storage module is used for carrying out distributed storage on the highway data with various different structures acquired by the acquisition module;

the data management module is used for performing data management on the data acquired by the acquisition module and the data stored by the storage module, wherein the data management comprises but is not limited to data resource catalog management, data quality management and data security management;

the data analysis module is used for matching a corresponding AI algorithm model according to the service requirements and in combination with the service scene, and performing data screening and feature extraction on the data stored by the storage module to obtain an analysis result;

the API service module is used for providing an interface for a third-party platform, managing registration, release and scheduling of various scene services, and monitoring and managing all running services;

the portal service module is integrated with an application system, data resources and network resources and is used for providing a uniform access interface for data;

and the operation and maintenance management module is used for monitoring the whole system and the performance, acquiring the comprehensive performance index of the system and providing detailed performance monitoring for various bottom layer resources used by the system.

2. The highway section-level data staging system according to claim 1, wherein the data source comprises basic data, external data and operational data, the basic data is data owned by a highway section company, the external data provides user macroscopic data information support for intelligent highway management and control, and the operational data is used for establishing an intelligent traffic management and control model algorithm, forming a highway section traffic early warning and management and control strategy and achieving assessment and management of traffic states and safety risks of a highway section.

3. The highway section-level data middling station system according to claim 1, wherein the multi-source data standardized interface adopts a Web Service technology, a SOAP transmission protocol is utilized, a data exchange interface specification is described by adopting an XML/json format, binary data is encoded by adopting a Base64 mode, and the data specification comprises uniform classified encoding, uniform data acquisition procedures, uniform data quality standards, uniform data formats and uniform metadata standards.

4. The highway section-level data staging system according to claim 1, wherein the storage module has a fully distributed architecture, data block granularity segmentation, online capacity expansion and reduction, copy backup and general PC hardware applicability technology, supports safe online storage of data of scale above PB level, and is further configured to store structured, semi-structured and unstructured data, and the storage module is further configured to store data in each data area according to data topics, where the data topics include, but are not limited to, project topics and tag library topics.

5. The highway section-level data staging system according to claim 1, wherein the data resource catalog management includes catalog registration, catalog update, metadata synchronization, master data synchronization, catalog inquiry, data archiving and recovery, the data quality management includes data quality inspection rules, data problem presentation, task management, link guarantee, environment and data quality monitoring and periodic reporting for third party platforms, the data security management includes database state monitoring, database audit, database risk scanning and access control, and desensitization and deformation are performed on sensitive information by hierarchical construction and hierarchical protection to solve data desensitization, traceability and audit problems.

6. The highway section-level data staging system according to claim 1, wherein the data analysis module is further configured to, after screening the data stored in the storage module, pre-process the screened data, and the pre-process includes: and performing set calibration on the screened data, images and images, and converting the data format.

7. The highway segment-level data staging system of claim 1, wherein the feature extraction comprises: calculating the gain value of each screened data, sequencing the gain values of the screened data from large to small, outputting the characteristic item with the maximum gain value to a file, vectorizing the original document and the document to be classified, calculating the cosine of an included angle between the original document and the document to be classified, selecting the text most similar to the new text in the training text set, and forming characteristic level fusion.

8. The highway section-level data staging system of claim 1 wherein the operation and maintenance management module includes monitoring of trunked networks, Hadoop systems, resource operations, Web services, database monitoring, hardware devices, system loads, and early warning alarms and log statistical analysis.

Images