CN113535846A - Big data platform and construction method thereof - Google Patents
Big data platform and construction method thereof Download PDFInfo
- Publication number
- CN113535846A CN113535846A CN202010313417.9A CN202010313417A CN113535846A CN 113535846 A CN113535846 A CN 113535846A CN 202010313417 A CN202010313417 A CN 202010313417A CN 113535846 A CN113535846 A CN 113535846A
- Authority
- CN
- China
- Prior art keywords
- data
- platform
- management
- big data
- management system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000010276 construction Methods 0.000 title claims description 8
- 238000007726 management method Methods 0.000 claims abstract description 107
- 238000000034 method Methods 0.000 claims abstract description 25
- 238000013523 data management Methods 0.000 claims abstract description 21
- 238000013079 data visualisation Methods 0.000 claims abstract description 19
- 238000011161 development Methods 0.000 claims abstract description 16
- 230000008569 process Effects 0.000 claims abstract description 10
- 230000010354 integration Effects 0.000 claims abstract description 7
- 230000004044 response Effects 0.000 claims abstract description 6
- 238000012544 monitoring process Methods 0.000 claims description 24
- 230000006870 function Effects 0.000 claims description 15
- 238000012795 verification Methods 0.000 claims description 14
- 238000010586 diagram Methods 0.000 claims description 11
- 238000013461 design Methods 0.000 claims description 8
- 238000004140 cleaning Methods 0.000 claims description 7
- 230000007246 mechanism Effects 0.000 claims description 7
- 238000004458 analytical method Methods 0.000 claims description 6
- 238000009826 distribution Methods 0.000 claims description 6
- 238000007619 statistical method Methods 0.000 claims description 6
- 238000005516 engineering process Methods 0.000 claims description 5
- 238000005259 measurement Methods 0.000 claims description 5
- 238000013441 quality evaluation Methods 0.000 claims description 5
- 230000005540 biological transmission Effects 0.000 claims description 4
- 238000013480 data collection Methods 0.000 claims description 4
- 238000001914 filtration Methods 0.000 claims description 4
- 238000005520 cutting process Methods 0.000 claims description 3
- 238000007405 data analysis Methods 0.000 claims description 3
- 238000005553 drilling Methods 0.000 claims description 3
- 239000000284 extract Substances 0.000 claims description 3
- 210000001503 joint Anatomy 0.000 claims description 3
- 238000002372 labelling Methods 0.000 claims description 3
- 238000012423 maintenance Methods 0.000 claims description 3
- 238000013508 migration Methods 0.000 claims description 3
- 230000005012 migration Effects 0.000 claims description 3
- 239000010453 quartz Substances 0.000 claims description 3
- VYPSYNLAJGMNEJ-UHFFFAOYSA-N silicon dioxide Inorganic materials O=[Si]=O VYPSYNLAJGMNEJ-UHFFFAOYSA-N 0.000 claims description 3
- 238000012800 visualization Methods 0.000 claims description 3
- 238000012545 processing Methods 0.000 description 13
- 238000003860 storage Methods 0.000 description 8
- 238000000586 desensitisation Methods 0.000 description 4
- 230000005180 public health Effects 0.000 description 4
- 238000013075 data extraction Methods 0.000 description 3
- 238000013500 data storage Methods 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 238000003032 molecular docking Methods 0.000 description 1
- 238000013439 planning Methods 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2462—Approximate or statistical queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/29—Geographical information databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/30—Authentication, i.e. establishing the identity or authorisation of security principals
- G06F21/31—User authentication
- G06F21/33—User authentication using certificates
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/604—Tools and structures for managing or administering access control systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
- G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
- G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
- G06F21/6245—Protecting personal data, e.g. for financial or medical purposes
- G06F21/6254—Protecting personal data, e.g. for financial or medical purposes by anonymising data, e.g. decorrelating personal data from the owner's identification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/547—Remote procedure calls [RPC]; Web services
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/10—Services
- G06Q50/26—Government or public services
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2221/00—Indexing scheme relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F2221/21—Indexing scheme relating to G06F21/00 and subgroups addressing additional information or applications relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F2221/2141—Access rights, e.g. capability lists, access control lists, access tables, access matrices
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A30/00—Adapting or protecting infrastructure or their operation
- Y02A30/60—Planning or developing urban green infrastructure
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Computer Security & Cryptography (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Business, Economics & Management (AREA)
- Computer Hardware Design (AREA)
- Bioethics (AREA)
- Tourism & Hospitality (AREA)
- Probability & Statistics with Applications (AREA)
- Computational Linguistics (AREA)
- Remote Sensing (AREA)
- Quality & Reliability (AREA)
- Mathematical Physics (AREA)
- Medical Informatics (AREA)
- Automation & Control Theory (AREA)
- Fuzzy Systems (AREA)
- Development Economics (AREA)
- Educational Administration (AREA)
- Computing Systems (AREA)
- Economics (AREA)
- Human Resources & Organizations (AREA)
- Marketing (AREA)
- Primary Health Care (AREA)
- Strategic Management (AREA)
- General Business, Economics & Management (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a big data platform, which comprises nine subsystems: the system comprises a data base platform library, a data exchange management system, a data sharing management system, a data management system, an operation support management system, a unified data portal management system, a big data visualization platform and a city operation map module. The method and the system can effectively help the third-party application to joint government affair units, reduce the repeated development of the access application and the supporting platform, realize the integration and sharing of resources and solve the problem of information isolated island. The invention uses the distributed database and is assisted with technical means such as a distributed file system HDFS, a MapReduce and the like to process and analyze a large amount of data in real time, thereby greatly optimizing the system performance, increasing various access interfaces, providing convenience for using data for a platform developer, and being generally suitable for service scenes with high concurrent access requirements, strong transaction consistency and low transaction response delay.
Description
Technical Field
The invention relates to the field of government affair data management, in particular to a big data platform and a construction method thereof.
Background
Among a plurality of government departments, a large amount of structured and unstructured data are generated, including public security population, enterprise and vehicle data, social security, labor and other data of people and society, sanitation and family planning data of a health care, examination and approval and a large amount of document data generated in work, and massive video data generated by law enforcement, so that massive data which have important influence on the aspects of the city are formed. The data are often stored in self-built systems of various departments, and the problems of difficult technical docking, data leakage and the like exist in data sharing.
The big data platform is developed to collect, clean and convert information resource data of all departments in order to solve the problems, and adopts different data exchange and integration modes according to different network environments and different data types to realize data aggregation.
The big data platform is an architecture platform which is in compliance with the development of the current informatization technology level and serves the reform of government functions. The method mainly aims to establish a government affair unified big data platform, realize safe sharing of government affair data, establish a standardized government affair data management system, promote sharing of government affair data and business cooperation, provide timely, accurate and reliable information basis for decision making, improve the foresight and pertinence of government affair work, increase the macroscopic regulation and control force and promote the sustainable and healthy development of economy.
The existing big data Platform in the market is mostly developed based on J2EE Java 2Platform Enterprise Edition (Java 2Platform, Enterprise Edition). J2EE is a set of technical architecture completely different from the traditional application development, and includes many components, which can simplify and standardize the development and deployment of the application system, thereby improving the portability, safety and reuse value. The core of the J2EE is a set of technical specifications and guidelines, wherein various components, service architectures and technical layers contained in the guidelines all have common standards and specifications, so that various platforms conforming to the J2EE architecture have good compatibility, and the dilemma that the information products used at the back end of an enterprise cannot be compatible with each other in the past, which causes the internal or external of the enterprise to be difficult to communicate is solved.
Among them, SSM is a development framework integrated using J2EE technology, namely the integrated Spring MVC, Spring, and Mybatis framework. It provides a fully functional MVC module that builds Web applications, separating the roles of controllers, model objects, filters, and handler objects, which makes them easier to customize.
Spring is a lightweight open-source framework, has the main characteristics of convenient decoupling, simplified development, section-oriented (AOP) programming support and declarative transaction support, and has the main advantages of low-intrusive design, independence from an application server and capability of performing centralized processing on common tasks such as logs and the like. Mybatis is a lightweight ORM framework that eliminates the manual setup of almost all JDBC code and parameters and retrieval of result sets, mapping interfaces and Java pojs to records in a database using simple XML or annotations for configuration and raw mapping. The frame structure diagram is shown in fig. 1.
The advantages are that: compared with the traditional SSH integration, SpringMVC is lighter than Struts, development consumption is reduced due to the use of annotations, and the database operation layer adopts Mybatis and can uniformly manage SQL.
A general big data platform in the market is a web application developed based on J2EE, and basically achieves functions of data collection, sorting, cleaning, sharing and the like, but due to the fact that a large amount of structured and unstructured data processing, the response time of a server is too long during web query, a browser is falsely dead, and the server is crashed due to too much pressure during data cleaning and filtering.
Because of the manner in which conventional web applications use a single database store, the processing of large amounts of structured and unstructured data by a single database can result in inefficient data processing due to the inability of database server performance to process large amounts of data simultaneously.
Therefore, it is necessary to provide a new big data platform to solve the above problems.
Disclosure of Invention
The invention aims to overcome the defects of the prior art, provides a large data platform, can effectively help a third-party application to joint government affairs units, reduces repeated development of an access application and a supporting platform, realizes integration and sharing of resources and solves the problem of information isolated island. In order to solve the problem of low performance caused by the fact that a traditional web big data platform uses a single database for storage, the big data platform uses a distributed database and is assisted with technical means such as a distributed file system (HDFS) and a MapReduce to process and analyze a large amount of data in real time. The system performance is greatly optimized, various access interfaces are added, data are conveniently used by a platform developer, and the method is generally suitable for service scenes with high concurrent access requirements, strong transaction consistency and low transaction response delay.
The invention further aims to provide a construction method of the big data platform.
The purpose of the invention is realized by the following technical scheme:
a big data platform comprises a data base platform library, a data exchange management system, a data sharing management system, a data governance management system, an operation support management system, a unified data portal management system, a big data visualization platform and a city operation map module; wherein
The data base platform library comprises a population information library, a legal person information library, a basic geographic space library, a macro economic library, a credit information library and an urban building information library;
the data exchange management system collects the data resources of each commission office to a big data platform through a big data exchange component according to unified standards and specifications, so as to realize the convergence and transmission of information resources; the requirements of each agency on transverse exchange of real-time information, business cooperation and the like are met, and information exchange and sharing services are provided for government affair cooperation, public services, auxiliary decisions and the like;
the data management system adopts an SOA architecture, custom releases the data of the heterogeneous database into WebService service, and provides for development and calling of an application system;
the data sharing management system comprises a data directory and a data API: the data directory component collects description information of data stored in each database by providing a uniform directory data publishing interface and standard metadata specifications, and establishes an information resource directory, so that the information resource directory is used as a way for various users to search legal persons, natural persons, spatial information, macroscopic economic information and thematic information, discovery and positioning of information resources are realized, and the users can conveniently and rapidly inquire, browse and apply for required data; the data API provides a flexible, convenient and quick data API management tool, and the sharing and use of data become clearer and more clear. A user can apply for issuing customized data API by self and call different types of data, and meanwhile, the platform records the interface calling condition and can realize an auxiliary management function, wherein the auxiliary management function comprises analyzing the commonly used data calling content;
the data management system provides full life cycle management, supports data standard establishment, data quality management, data asset management and data analysis of the data management capacity of the whole full life cycle; the task scheduling and monitoring capability of the data whole life process is provided; providing statistical analysis capabilities for the data assets;
the data management system has a perfect data safety mechanism, provides all-direction data cleaning, verifying and encrypting capabilities, and ensures the consistency, integrity, uniqueness and accuracy of data; the configurable measurement rule and verification method generation capability is provided, and the flexible scheduling execution capability is provided; and flexibly generating a data quality evaluation report. The access security control of data and resource levels is supported, and desensitization data can be flexibly managed;
the operation support management system comprises a single sign-on module and a background management module; the Single Sign On (SSO for short) module is combined with the CA/PKI system, so that in a plurality of application systems, a user can access all mutually trusted application systems only by logging in once, and the purposes of one-time logging and free-roaming all over the network are achieved; the background management module provides a function of configuring the authority of the big data platform for background management personnel;
the unified data portal management system is authorized by the management authority in a classified and graded mode and is divided into a commission office data manager, a commission office data user, an information office manager, an information office user, operation and maintenance personnel, a data analyzer and a general manager according to the user category;
the big data visualization platform is used for providing visualization graph display capability of big data;
the city operation map module focuses on the comprehensive management and operation condition monitoring capability of cities, aims at the overall coordinated development and high-efficiency operation of the cities, converges data in aspects of important project management, city environment monitoring, city sound monitoring, city basic resource management, city safety supervision management, education service, public safety management, public health management and the like, realizes real-time and high-efficiency monitoring on city management through visual display, prediction and analysis on the city operation condition, provides basic support for leadership decision making, improves the city operation management efficiency, improves the quality level of civil service, and provides quick, high-quality and high-efficiency comprehensive service for citizens and governments.
The geographic information-map platform focuses on the comprehensive management and operation condition monitoring capability of cities, aims at the overall coordinated development and high-efficiency operation of the cities, converges data in aspects of important project management, urban environment monitoring, urban sound monitoring, urban basic resource management, urban safety supervision management, education service, public safety management, public health management and the like, visually displays the overall operation condition of the cities through the data processed by the big data platform (for example, displays the point positions of recreation and entertainment places in the whole city on an urban map, and makes targeted management by visually knowing the distribution condition of the centers of the city residents for life and entertainment), realizes the real-time and high-efficiency monitoring of the urban management, provides basic support for leadership decision making, improves the urban operation management efficiency, improves the quality level of the civil service, and provides rapid monitoring for citizens and governments, High-quality and high-efficiency comprehensive service.
The data management system manages the data quality in the big data platform through an ETL tool, and the data ETL periodically extracts, converts and loads data from each data source to the big data platform; data asset management is carried out on data in the big data platform by tracing and recording the trace and the source of the data into a big data platform supporting database, SQL sentences are used for analyzing the data in the whole life cycle of the data to extract high-quality data, thereby providing the statistical analysis capability of the data assets and flexibly generating a data quality evaluation report; the method has the advantages that the configuration measurement rule and verification method generation capacity is provided by using Quartz in combination with spring on a large data platform, the flexible scheduling execution capacity is provided, and the data has task scheduling and monitoring capacity in the whole life process.
The data management and management system uses the sugondc to carry out all-around data cleaning, verification and encryption. Therefore, the big data platform has a perfect data security mechanism, and the consistency, integrity, uniqueness and accuracy of data are guaranteed; the big data platform supports data and resource level access security control on a system level, and can flexibly manage desensitization data.
In the operation support management system, the PKI system is the combination of computer software and hardware, an authority mechanism and an application system; it provides user authentication service for implementing e-commerce, e-government affairs, office automation, etc.
The PKI system comprises a Key Management Center (KMC), a Certification Authority (CA), a registration auditing Authority (RA), a Certificate issuing system and an application interface, and is based on a public key technology, the public key is managed by adopting a Certificate, and the public key of a user and the identity of the user are bound together through a trusted third party certification Center (CA).
In the operation support management system, a user accesses a target system through a browser, and the target system subsequently requests an SSO server (namely a single sign-on module server) and logs in by using a key; the SSO server authenticates the identity according to a request (including a user digital certificate) and calls a login request, wherein the login request comprises a target system address, the login request comprises creation time, a random number, SSO server signature information and the user digital certificate; after the request is successful, returning the target system URL to the client browser, verifying the validity of the login request by the SSO verification module and obtaining other parameters, if the login request passes the verification, analyzing the login request, reading the user information and verifying the user access authority; setting session information, wherein the session information comprises user information and an expiration date. The background management subsystem provides a function of configuring the authority of the big data platform for background management personnel.
The big data visualization platform is used for realizing butt joint of data sources to be displayed, has conditional query and filtering capabilities on accessed data, supports OLAP manipulation of multi-dimensional data, supports operations of scrolling, drilling, slicing, cutting into blocks and rotating on data cubes, and supports dynamic drawing and dynamic refreshing of data display; through a graphical design and arrangement tool, a data display interface is freely designed in a dragging mode based on a graphical environment, and a graphical display component is supported; the GIS components and the GIS-based spatial display of all service data are supported; an application that provides a vector map; the control event response capability of each graphic component can be realized, and the event correlation among the components is supported; the data display constructed based on the big data visualization platform can support big screen data display and data display of a WEB portal;
in the big data visualization platform, the graphical display components comprise a line graph, a bar graph, an area graph, a pie graph, a pile-up graph, a radar graph, a bubble graph, a scatter graph, a chord graph, a force guidance relation graph, a filled bubble graph, an Asahi day graph, a video component and a calendar component.
In the big data visualization platform, the vector map comprises a distribution map, a migration map, a labeling map and a thermodynamic diagram.
The other purpose of the invention is realized by the following technical scheme:
a big data platform construction method comprises the following steps:
s1, data collection: collecting data owned by each committee;
s2, data combing and analyzing: combing data by using an ETL tool and an SQL statement, and performing relevance analysis on the data to form the demand and sharing of a data resource catalog;
s3, data integration: using SQL sentences to converge and integrate data resources of different departments and different types according to the data catalog to form a big data set;
s4, sharing and exchanging data: a user applies for required data by logging in a big data platform, and the applied object is a conditional shared directory or an unconditional shared directory; and the administrator agrees to open the data after applying, so that the data is shared and exchanged, and the data circulation is realized.
Compared with the prior art, the invention has the following advantages and beneficial effects:
1. the application of the distributed file system HDFS and the data operation model MapReduce used by the big data platform is a key point and a point to be protected.
2. The performance of the existing big data platform in the market cannot be compared with that of a big data platform adopting distributed storage HDFS due to the adoption of a single database storage mode, and the government big data platform adopting the HDFS can write one hundred million data per second and provides an interface for increasing the memory at any time for the platform, so that the capacity of the system can be expanded as required.
3. The big data platform of the invention uses a data operation model MapReduce, so that the system has the capacity of processing big data sets. The user model can be calculated in real time, the user behavior can be predicted and analyzed, and a decision analysis basis is provided for leaders using a government big data platform.
4. The big data platform has safer data extraction, higher data storage performance and quicker data operation. The data extraction safety of the big data platform is guaranteed in a software and hardware cooperative work mode, and guarantees are provided for data extraction, sharing and the like. The distributed storage HDFS is used, and more than one hundred million data can be easily written in each second when the data throughput is very high. The data operation model MapReduce can perform data operation on a large amount of data in real time.
Drawings
Fig. 1 is a frame structure diagram of Spring.
Fig. 2 is an operation flowchart of the operation support management system.
FIG. 3 is a block diagram of a software architecture for a big data platform.
Fig. 4 is a sharing flow diagram of conditional sharing.
Fig. 5 is a sharing flow chart of unconditional sharing.
FIG. 6 is a design diagram of a big data platform for implementing data sharing exchange, and a specific data processing and storage system.
FIG. 7 is a schematic diagram of a distributed data storage process.
Fig. 8 is a schematic structural diagram of a combination of software and hardware of a big data platform.
Detailed Description
The present invention will be described in further detail with reference to examples and drawings, but the present invention is not limited thereto.
A big data platform comprises a data base platform library, a data exchange management system, a data sharing management system, a data governance management system, an operation support management system, a unified data portal management system, a big data visualization platform and a city operation map module; wherein
The data base platform library comprises a population information library, a legal person information library, a basic geographic space library, a macro economic library, a credit information library and an urban building information library;
the data exchange management system collects the data resources of each commission office to a big data platform through a big data exchange component according to unified standards and specifications, so as to realize the convergence and transmission of information resources; the requirements of each agency on transverse exchange of real-time information, business cooperation and the like are met, and information exchange and sharing services are provided for government affair cooperation, public services, auxiliary decisions and the like;
the data management system adopts an SOA architecture, custom releases the data of the heterogeneous database into WebService service, and provides for development and calling of an application system;
the data sharing management system comprises a data directory and a data API: the data directory component collects description information of data stored in each database by providing a uniform directory data publishing interface and standard metadata specifications, and establishes an information resource directory, so that the information resource directory is used as a way for various users to search legal persons, natural persons, spatial information, macroscopic economic information and thematic information, discovery and positioning of information resources are realized, and the users can conveniently and rapidly inquire, browse and apply for required data; the data API provides a flexible, convenient and quick data API management tool, and the sharing and use of data become clearer and more clear. A user can apply for issuing customized data API by self and call different types of data, and meanwhile, the platform records the interface calling condition and can realize an auxiliary management function, wherein the auxiliary management function comprises analyzing the commonly used data calling content;
the data management system provides full life cycle management, supports data standard establishment, data quality management, data asset management and data analysis of the data management capacity of the whole full life cycle; the task scheduling and monitoring capability of the data whole life process is provided; providing statistical analysis capabilities for the data assets;
the data management system has a perfect data safety mechanism, provides all-direction data cleaning, verifying and encrypting capabilities, and ensures the consistency, integrity, uniqueness and accuracy of data; the configurable measurement rule and verification method generation capability is provided, and the flexible scheduling execution capability is provided; and flexibly generating a data quality evaluation report. The access security control of data and resource levels is supported, and desensitization data can be flexibly managed;
the operation support management system comprises a single sign-on module and a background management module; the Single Sign On (SSO for short) module is combined with the CA/PKI system, so that in a plurality of application systems, a user can access all mutually trusted application systems only by logging in once, and the purposes of one-time logging and free-roaming all over the network are achieved; the background management module provides a function of configuring the authority of the big data platform for background management personnel;
the unified data portal management system is authorized by the management authority in a classified and graded mode and is divided into a commission office data manager, a commission office data user, an information office manager, an information office user, operation and maintenance personnel, a data analyzer and a general manager according to the user category;
the big data visualization platform is used for providing visualization graph display capability of big data;
the city operation map module focuses on the comprehensive management and operation condition monitoring capability of cities, aims at the overall coordinated development and high-efficiency operation of the cities, converges data in aspects of important project management, city environment monitoring, city sound monitoring, city basic resource management, city safety supervision management, education service, public safety management, public health management and the like, realizes real-time and high-efficiency monitoring on city management through visual display, prediction and analysis on the city operation condition, provides basic support for leadership decision making, improves the city operation management efficiency, improves the quality level of civil service, and provides quick, high-quality and high-efficiency comprehensive service for citizens and governments.
The geographic information-map platform focuses on the comprehensive management and operation condition monitoring capability of cities, aims at the overall coordinated development and high-efficiency operation of the cities, converges data in aspects of important project management, urban environment monitoring, urban sound monitoring, urban basic resource management, urban safety supervision management, education service, public safety management, public health management and the like, visually displays the overall operation condition of the cities through the data processed by the big data platform (for example, displays the point positions of recreation and entertainment places in the whole city on an urban map, and makes targeted management by visually knowing the distribution condition of the centers of the city residents for life and entertainment), realizes the real-time and high-efficiency monitoring of the urban management, provides basic support for leadership decision making, improves the urban operation management efficiency, improves the quality level of the civil service, and provides rapid monitoring for citizens and governments, High-quality and high-efficiency comprehensive service.
FIG. 3 is a block diagram of a big data platform software architecture.
The data management system manages the data quality in the big data platform through an ETL tool, and the data ETL periodically extracts, converts and loads data from each data source to the big data platform; data asset management is carried out on data in the big data platform by tracing and recording the trace and the source of the data into a big data platform supporting database, SQL sentences are used for analyzing the data in the whole life cycle of the data to extract high-quality data, thereby providing the statistical analysis capability of the data assets and flexibly generating a data quality evaluation report; the method has the advantages that the configuration measurement rule and verification method generation capacity is provided by using Quartz in combination with spring on a large data platform, the flexible scheduling execution capacity is provided, and the data has task scheduling and monitoring capacity in the whole life process.
The data management and management system uses the sugondc to carry out all-around data cleaning, verification and encryption. Therefore, the big data platform has a perfect data security mechanism, and the consistency, integrity, uniqueness and accuracy of data are guaranteed; the big data platform supports data and resource level access security control on a system level, and can flexibly manage desensitization data.
In the operation support management system, the PKI system is the combination of computer software and hardware, an authority mechanism and an application system; it provides user authentication service for implementing e-commerce, e-government affairs, office automation, etc.
The PKI system comprises a Key Management Center (KMC), a Certification Authority (CA), a registration auditing Authority (RA), a Certificate issuing system and an application interface, and is based on a public key technology, the public key is managed by adopting a Certificate, and the public key of a user and the identity of the user are bound together through a trusted third party certification Center (CA).
In the operation support management system, a user accesses a target system through a browser, and the target system subsequently requests an SSO server (namely a single sign-on module server) and logs in by using a key; the SSO server authenticates the identity according to a request (including a user digital certificate) and calls a login request, wherein the login request comprises a target system address, the login request comprises creation time, a random number, SSO server signature information and the user digital certificate; after the request is successful, returning the target system URL to the client browser, verifying the validity of the login request by the SSO verification module and obtaining other parameters, if the login request passes the verification, analyzing the login request, reading the user information and verifying the user access authority; setting session information, wherein the session information comprises user information and an expiration date. The overall function is shown in figure 2. The background management subsystem provides a function of configuring the authority of the big data platform for background management personnel.
The big data visualization platform is used for realizing butt joint of data sources to be displayed, has conditional query and filtering capabilities on accessed data, supports OLAP manipulation of multi-dimensional data, supports operations of scrolling, drilling, slicing, cutting into blocks and rotating on data cubes, and supports dynamic drawing and dynamic refreshing of data display; through a graphical design and arrangement tool, a data display interface is freely designed in a dragging mode based on a graphical environment, and a graphical display component is supported; the GIS components and the GIS-based spatial display of all service data are supported; an application that provides a vector map; the control event response capability of each graphic component can be realized, and the event correlation among the components is supported; the data display constructed based on the big data visualization platform can support big screen data display and data display of a WEB portal;
in the big data visualization platform, the graphical display components comprise a line graph, a bar graph, an area graph, a pie graph, a pile-up graph, a radar graph, a bubble graph, a scatter graph, a chord graph, a force guidance relation graph, a filled bubble graph, an Asahi day graph, a video component and a calendar component.
In the big data visualization platform, the vector map comprises a distribution map, a migration map, a labeling map and a thermodynamic diagram.
A big data platform construction method comprises the following steps:
s1, data collection: collecting data owned by each committee;
s2, data combing and analyzing: combing data by using an ETL tool and an SQL statement, and performing relevance analysis on the data to form the demand and sharing of a data resource catalog;
s3, data integration: using SQL sentences to converge and integrate data resources of different departments and different types according to the data catalog to form a big data set;
s4, sharing and exchanging data: a user applies for required data by logging in a big data platform, and the applied object is a conditional shared directory or an unconditional shared directory; and the administrator agrees to open the data after applying, so that the data is shared and exchanged, and the data circulation is realized.
In step S4, the sharing of the data is switched to conditional sharing (the subject of application is a conditional sharing directory), and the flow thereof is as shown in fig. 4.
In step S4, the sharing of the data is switched to unconditional sharing (the application target is an unconditional shared directory), and the flow thereof is as shown in fig. 5.
As shown in fig. 6, the bottom layer of the big data platform uses a distributed file system to solve the problem of low processing efficiency of a large amount of structured and unstructured data, and uses MapReduce as a data operation model to perform parallel operation on a large-scale data set (larger than 1TB), thereby breaking through the performance bottleneck of the traditional web big data platform.
The distributed file system HDFS is used as a distributed storage software framework for storing mass data, is suitable for a distributed file system running on general hardware, has the characteristics of high fault tolerance, high throughput data access and the like, and is suitable for being applied to a large-scale data set. The distributed file storage has the following characteristics:
1. and (3) cluster service deployment: the method supports massive host resource clusters and supports the deployment of environments on a universal and cheap server;
2. fault tolerance and stability of the system: in the face of a large number of nodes and file structures in a file system, any group of files can be invalid, so that the platform supports error detection and rapid and automatic recovery;
3. stream data access: applications running on HDFS, unlike ordinary applications, require streaming access to their data sets. The design of the HDFS takes data batch processing and large file operation into more consideration;
4. high throughput data access: HDFS relaxes a portion of the POSIX constraints to achieve the goal of streaming file system data. More critical than the low latency problem of data access is the high throughput of data access;
5. large-scale data set: the HDFS supports large file storage, can provide high data transmission bandwidth, and can be expanded to hundreds of nodes in a cluster. A single HDFS instance should be able to support tens of millions of files
6. Simple consistency model: HDFS applications require a "write-once read-many" file access model. A file does not need to be changed after it is created, written, and closed. This assumption simplifies the data consistency problem and enables high throughput data access;
7. cluster balancing load: and supporting a data balancing strategy, and if the free space on a certain node is lower than a critical point, automatically moving data among the nodes according to the balancing strategy by a system. When a request for a file suddenly increases, a plan may also be initiated to create a new copy of the file and at the same time rebalance other data in the cluster.
8. File system interface: a variety of access interfaces are provided, such as Java API access interfaces.
As shown in fig. 7, MapReduce is a computation model, framework and platform oriented to parallel processing of big data, and implies the following three layers:
MapReduce is a Cluster-based high-performance parallel computing platform (Cluster InfraStrustStructure). It allows the construction of a distributed and parallel computing cluster containing tens, hundreds or thousands of nodes with commercially available servers.
MapReduce is a parallel computing and running Software Framework (Software Framework). The parallel computing software framework is huge but has a fine design, can automatically complete the parallel processing of computing tasks, automatically divide computing data and computing tasks, automatically distribute and execute the tasks on cluster nodes and collect computing results, and sends many complex details at the bottom of the system related to the parallel computing such as data distribution storage, data communication, fault-tolerant processing and the like to the system for processing, thereby greatly reducing the burden of software developers.
MapReduce is a parallel Programming Model & method. The method provides a simple and convenient parallel program design method by means of the design idea of a functional programming language Lisp, realizes basic parallel computing tasks by using two functions of Map and Reduce, provides abstract operation and a parallel programming interface, and simply and conveniently completes the programming and computing processing of large-scale data.
As shown in fig. 8, the big data platform uses a combination of software and hardware to improve the data use security.
The above embodiments are preferred embodiments of the present invention, but the present invention is not limited to the above embodiments, and any other changes, modifications, substitutions, combinations, and simplifications which do not depart from the spirit and principle of the present invention should be construed as equivalents thereof, and all such changes, modifications, substitutions, combinations, and simplifications are intended to be included in the scope of the present invention.
Claims (10)
1. A big data platform, comprising: the system comprises a data base platform library, a data exchange management system, a data sharing management system, a data management system, an operation support management system, a unified data portal management system, a big data visualization platform and a city operation map module; wherein,
the data base platform library comprises a population information library, a legal person information library, a basic geographic space library, a macro economic library, a credit information library and an urban building information library;
the data exchange management system collects the data resources of each commission office to a big data platform through a big data exchange component according to unified standards and specifications, so as to realize the convergence and transmission of information resources;
the data management system adopts an SOA architecture, custom releases the data of the heterogeneous database into WebService service, and provides for development and calling of an application system;
the data sharing management system comprises a data directory and a data API: the data directory component collects the description information of the data stored in each database by providing a uniform directory data release interface and standard metadata specifications, and establishes an information resource directory, so that the information resource directory is used as a way for various users to search legal persons, natural persons, spatial information, macroscopic economic information and thematic information, and discovery and positioning of information resources are realized; a user can apply for issuing customized data API by self and call different types of data, and meanwhile, the platform records the interface calling condition and can realize an auxiliary management function, wherein the auxiliary management function comprises analyzing the commonly used data calling content;
the data management system provides full life cycle management, supports data standard establishment, data quality management, data asset management and data analysis of the data management capacity of the whole full life cycle; the task scheduling and monitoring capability of the data whole life process is provided; providing statistical analysis capabilities for the data assets;
the operation support management system comprises a single sign-on module and a background management module; the single sign-on module is combined with a CA/PKI system, so that in a plurality of application systems, a user can access all mutually trusted application systems only by logging on once; the background management module provides a function of configuring the authority of the big data platform for background management personnel;
the unified data portal management system is authorized by the management authority in a classified and graded mode and is divided into a commission office data manager, a commission office data user, an information office manager, an information office user, operation and maintenance personnel, a data analyzer and a general manager according to the user category;
the big data visualization platform is used for providing visualization graph display capability of big data;
and the city runs a graph module, and the whole running condition of the city is visually displayed through the data processed by the big data platform.
2. The big data platform of claim 1, wherein: the data management system manages the data quality in the big data platform through an ETL tool, and the data ETL periodically extracts, converts and loads data from each data source to the big data platform; data asset management is carried out on data in the big data platform by tracing and recording the trace and the source of the data into a big data platform supporting database, SQL sentences are used for analyzing the data in the whole life cycle of the data to extract high-quality data, thereby providing the statistical analysis capability of the data assets and generating a data quality evaluation report; the method has the advantages that the configuration measurement rule and verification method generation capacity is provided by using Quartz in combination with spring on a large data platform, the flexible scheduling execution capacity is provided, and the data has task scheduling and monitoring capacity in the whole life process.
3. The big data platform of claim 1, wherein: the data management and management system uses the sugondc to carry out all-around data cleaning, verification and encryption.
4. The big data platform of claim 1, wherein: in the operation support management system, the PKI system is the combination of computer software and hardware, an authority mechanism and an application system.
5. The big data platform of claim 4, wherein: the PKI system comprises a key management center, a certification authority, a registration auditing authority, a certificate issuing system and an application interface, and is based on a public key technology, the public key is managed by adopting a certificate, and the public key of a user and the identity of the user are bound together through a trusted third-party certification center.
6. The big data platform of claim 1, wherein: in the operation support management system, a user accesses a target system through a browser, and the target system then requests an SSO server and logs in by using a key; the SSO server authenticates the identity according to the request and calls a login request, wherein the login request comprises a target system address, the login request comprises creation time, a random number, SSO server signature information and a digital certificate of a user; after the request is successful, returning the target system URL to the client browser, verifying the validity of the login request by the SSO verification module and obtaining other parameters, if the login request passes the verification, analyzing the login request, reading the user information and verifying the user access authority; setting session information, wherein the session information comprises user information and an expiration date.
7. The big data platform of claim 1, wherein: the big data visualization platform is used for realizing butt joint of data sources to be displayed, has conditional query and filtering capabilities on accessed data, supports OLAP manipulation of multi-dimensional data, supports operations of scrolling, drilling, slicing, cutting into blocks and rotating on data cubes, and supports dynamic drawing and dynamic refreshing of data display; through a graphical design and arrangement tool, a data display interface is freely designed in a dragging mode based on a graphical environment, and a graphical display component is supported; the GIS components and the GIS-based spatial display of all service data are supported; an application that provides a vector map; the control event response capability of each graphic component can be realized, and the event correlation among the components is supported; the data display constructed based on the big data visualization platform can support big screen data display and data display of a WEB portal.
8. The big data platform of claim 7, wherein: in the big data visualization platform, the graphical display components comprise a line graph, a bar graph, an area graph, a pie graph, a pile-up graph, a radar graph, a bubble graph, a scatter graph, a chord graph, a force guidance relation graph, a filled bubble graph, an Asahi day graph, a video component and a calendar component.
9. The big data platform of claim 7, wherein: in the big data visualization platform, the vector map comprises a distribution map, a migration map, a labeling map and a thermodynamic diagram.
10. A big data platform construction method is characterized by comprising the following steps:
s1, data collection: collecting data owned by each committee;
s2, data combing and analyzing: combing data by using an ETL tool and an SQL statement, and performing relevance analysis on the data to form the demand and sharing of a data resource catalog;
s3, data integration: using SQL sentences to converge and integrate data resources of different departments and different types according to the data catalog to form a big data set;
s4, sharing and exchanging data: a user applies for required data by logging in a big data platform, and the applied object is a conditional shared directory or an unconditional shared directory; and the administrator agrees to open the data after applying, so that the data is shared and exchanged, and the data circulation is realized.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010313417.9A CN113535846B (en) | 2020-04-20 | 2020-04-20 | Big data platform and construction method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010313417.9A CN113535846B (en) | 2020-04-20 | 2020-04-20 | Big data platform and construction method thereof |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113535846A true CN113535846A (en) | 2021-10-22 |
| CN113535846B CN113535846B (en) | 2023-08-08 |
Family
ID=78123650
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010313417.9A Active CN113535846B (en) | 2020-04-20 | 2020-04-20 | Big data platform and construction method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113535846B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114153912A (en) * | 2021-11-12 | 2022-03-08 | 上海通贸国际供应链管理有限公司 | Analysis method based on visual integration |
| CN114205116A (en) * | 2021-11-16 | 2022-03-18 | 广西中科曙光云计算有限公司 | Zero-trust borderless security access system |
| CN116414818A (en) * | 2023-06-09 | 2023-07-11 | 深圳市泰铼科技有限公司 | Distributed security data cleaning system based on visual management |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160055221A1 (en) * | 2014-08-19 | 2016-02-25 | Tagb.io. Inc. | Data Analysis And Visualization |
| CN105956112A (en) * | 2016-05-05 | 2016-09-21 | 云神科技投资股份有限公司 | Visual urban big data presentation platform and method |
| CN107846409A (en) * | 2017-11-17 | 2018-03-27 | 广州葵翼信息科技有限公司 | A kind of smart city network integration and safety management system |
| CN107945086A (en) * | 2017-11-17 | 2018-04-20 | 广州葵翼信息科技有限公司 | A kind of big data resource management system applied to smart city |
| KR101864222B1 (en) * | 2017-07-24 | 2018-06-04 | 엠제이코어텍(주) | system and method for pedestrian traffic big data analysis |
| CN108647217A (en) * | 2017-12-27 | 2018-10-12 | 广东智政信息科技有限公司 | Big data platform integrated management system based on safety supervision application |
| CN109522359A (en) * | 2018-11-02 | 2019-03-26 | 大连瀚闻资讯有限公司 | Visualization industrial analysis method based on big data |
| CN109711685A (en) * | 2018-12-14 | 2019-05-03 | 杨冰之 | A kind of government affairs big data processing platform |
| CN110781236A (en) * | 2019-10-29 | 2020-02-11 | 山西云时代技术有限公司 | Method for constructing government affair big data management system |
| CN110941749A (en) * | 2019-11-08 | 2020-03-31 | 苏州城方信息技术有限公司 | Visualization method for government affair big data full life cycle governance |
-
2020
- 2020-04-20 CN CN202010313417.9A patent/CN113535846B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160055221A1 (en) * | 2014-08-19 | 2016-02-25 | Tagb.io. Inc. | Data Analysis And Visualization |
| CN105956112A (en) * | 2016-05-05 | 2016-09-21 | 云神科技投资股份有限公司 | Visual urban big data presentation platform and method |
| KR101864222B1 (en) * | 2017-07-24 | 2018-06-04 | 엠제이코어텍(주) | system and method for pedestrian traffic big data analysis |
| CN107846409A (en) * | 2017-11-17 | 2018-03-27 | 广州葵翼信息科技有限公司 | A kind of smart city network integration and safety management system |
| CN107945086A (en) * | 2017-11-17 | 2018-04-20 | 广州葵翼信息科技有限公司 | A kind of big data resource management system applied to smart city |
| CN108647217A (en) * | 2017-12-27 | 2018-10-12 | 广东智政信息科技有限公司 | Big data platform integrated management system based on safety supervision application |
| CN109522359A (en) * | 2018-11-02 | 2019-03-26 | 大连瀚闻资讯有限公司 | Visualization industrial analysis method based on big data |
| CN109711685A (en) * | 2018-12-14 | 2019-05-03 | 杨冰之 | A kind of government affairs big data processing platform |
| CN110781236A (en) * | 2019-10-29 | 2020-02-11 | 山西云时代技术有限公司 | Method for constructing government affair big data management system |
| CN110941749A (en) * | 2019-11-08 | 2020-03-31 | 苏州城方信息技术有限公司 | Visualization method for government affair big data full life cycle governance |
Non-Patent Citations (2)
| Title |
|---|
| ALVARO GRAVES 等: "Visualization tools for open government data", 《DG.O \'13: PROCEEDINGS OF THE 14TH ANNUAL INTERNATIONAL CONFERENCE ON DIGITAL GOVERNMENT RESEARCH》, pages 136 - 145 * |
| 周君 等: "新型智慧城市下政务数据安全管理的研究", 信息通信技术与政策, no. 3, pages 32 - 36 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114153912A (en) * | 2021-11-12 | 2022-03-08 | 上海通贸国际供应链管理有限公司 | Analysis method based on visual integration |
| CN114205116A (en) * | 2021-11-16 | 2022-03-18 | 广西中科曙光云计算有限公司 | Zero-trust borderless security access system |
| CN114205116B (en) * | 2021-11-16 | 2023-12-19 | 广西中科曙光云计算有限公司 | Zero-trust borderless security access system |
| CN116414818A (en) * | 2023-06-09 | 2023-07-11 | 深圳市泰铼科技有限公司 | Distributed security data cleaning system based on visual management |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113535846B (en) | 2023-08-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112685385B (en) | Big data platform for smart city construction | |
| CN107315776B (en) | Data management system based on cloud computing | |
| CN113535846B (en) | Big data platform and construction method thereof | |
| Narkhede et al. | HMR log analyzer: Analyze web application logs over Hadoop MapReduce | |
| CN104363222A (en) | Hadoop-based network security event analysis method | |
| CN111930807B (en) | Rail transit data analysis method, device, equipment and storage medium | |
| CN118152481B (en) | Drug information storage method based on distributed edge calculation and multi-mode data | |
| Zheng | Database as a service-current issues and its future | |
| Zobaed et al. | Big Data in the Cloud. | |
| Michele et al. | Create dashboards and data story with the data & analytics frameworks | |
| Xiong et al. | Data vitalization's perspective towards smart city: a reference model for data service oriented architecture | |
| Martinviita | Time series database in Industrial IoT and its testing tool | |
| CN106993032A (en) | The embedded accurate communication cloud service platform applied based on mobile Internet | |
| Waseem et al. | Quantitative analysis and performance evaluation of target-oriented replication strategies in cloud computing | |
| CN113689175A (en) | Geographic information public service platform based on cross-platform architecture and construction method thereof | |
| Kukreja et al. | Data Engineering with Apache Spark, Delta Lake, and Lakehouse: Create scalable pipelines that ingest, curate, and aggregate complex data in a timely and secure way | |
| Zhang | Optimization Strategy of College Students’ Education Management Based on Smart Cloud Platform Teaching | |
| Xhafa et al. | Performance Evaluation of a MapReduce Hadoop-Based Implementation for Processing Large Virtual Campus Log Files | |
| Sun et al. | Design and Development of Commercial Bank Financial System Based on Cloud Computing | |
| Darius et al. | From Data to Insights: A Review of Cloud-Based Big Data Tools and Technologies | |
| Quintero et al. | IBM data engine for hadoop and spark | |
| Hu et al. | Implementation and Application of National Science and Technology Information System | |
| Li | Research Review of Cloud Computing Technology Based on Big Data | |
| Ma et al. | Evaluating distributed transactional database system | |
| Sharma et al. | Supportive architectural analysis for big data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |