KR20160075971A - Big data management system for public complaints services - Google Patents

Big data management system for public complaints services Download PDF

Info

Publication number
KR20160075971A
KR20160075971A KR1020140184749A KR20140184749A KR20160075971A KR 20160075971 A KR20160075971 A KR 20160075971A KR 1020140184749 A KR1020140184749 A KR 1020140184749A KR 20140184749 A KR20140184749 A KR 20140184749A KR 20160075971 A KR20160075971 A KR 20160075971A
Authority
KR
South Korea
Prior art keywords
data
public
module
civil
analysis
Prior art date
Application number
KR1020140184749A
Other languages
Korean (ko)
Inventor
남준
Original Assignee
케이웨어 (주)
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 케이웨어 (주) filed Critical 케이웨어 (주)
Priority to KR1020140184749A priority Critical patent/KR20160075971A/en
Publication of KR20160075971A publication Critical patent/KR20160075971A/en

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21Design, administration or maintenance of databases
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00Administration; Management
    • G06Q10/04Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem"
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q50/00Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
    • G06Q50/10Services
    • G06Q50/26Government or public services

Landscapes

  • Engineering & Computer Science (AREA)
  • Business, Economics & Management (AREA)
  • Theoretical Computer Science (AREA)
  • Strategic Management (AREA)
  • Economics (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Tourism & Hospitality (AREA)
  • Human Resources & Organizations (AREA)
  • Marketing (AREA)
  • General Business, Economics & Management (AREA)
  • Development Economics (AREA)
  • Databases & Information Systems (AREA)
  • Quality & Reliability (AREA)
  • General Engineering & Computer Science (AREA)
  • Operations Research (AREA)
  • Game Theory and Decision Science (AREA)
  • Data Mining & Analysis (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Educational Administration (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Primary Health Care (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

Big Data is rapidly becoming a world-wide concern for information technology, and focuses on what value public organizations and corporations will generate through the big data they have collected so far. Therefore, the big data management and system for the public service of the present invention constitutes the development step as follows. First, it is the development of the public complainant big data collector module which collects the complaint data provided by various sources in the public institution in real time on the web portal, SNS, and intranet within the public institution. Second, it is the development of a module for real-time storage management of public data that is stored in the Hadoop Distributed File System (HDFS) and stored in the relational database through real-time distributed parallel processing through the MapReduce framework. Third, it analyzes the data by theme using real-time data mining technology from stored public complaints data, and develops a public-private big data analysis and visualization processor module that predicts classification, grouping, and complaint trend. Finally, it is a public service system that can effectively provide civil service statistics, civil affairs policies, and civil service forecasts to improve the quality of public services.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0002] The present invention relates to a large data management system for public complaints data services,

The present invention relates to a big data management system for a public service data service for forecasting and visualizing services through data analysis as well as data collection, storage, processing, and management techniques in order to utilize big data of a public institution.

Big Data is rapidly becoming a world-wide concern for information technology, and focuses on what value public organizations and corporations will generate through the big data they have collected so far. Big Data Processing Analysis Hadoop, an open source project, is designed to make it easy to compile, query, and analyze large amounts of data, including Hadoop file system (HDFS), OS level abstractions and MapReduce engines HIVE, and Mahout, a distributed, parallel-capable machine learning library for developing intelligent applications that require large amounts of data. It also contains the necessary Java archive files (Java ARchive, JARs), scripts to start Hadoop, source code and related resources.

SUMMARY OF THE INVENTION The present invention has been developed in order to meet the above-described needs of the prior art. In the case of big data of a public complaint, the amount of data smaller than the data size is divided into several departments, not. Therefore, various approaches and analysis methods are needed for the big data of public complaints. The object of the present invention is to provide a big data management system for a public complaint data collector, a public complaint big data storage manager, a public complaint big data analysis and visualization processor, a public complaint statistics service and a public complaint data service, There is a purpose.

According to a first aspect of the present invention, there is provided a big data management system for a public service data service,

First, data is not linked because civil affairs data is managed by public institutions, local governments, and government departments. Therefore, it provides multi-source big data collection, storage management, analysis and visualization service considering various sources and formats for public application data.

Second, the government subsidized institutions developed services using some public application data, but there is no commercialization of civil service. Therefore, it provides convenience such that charts, graphs, and various visualization functions are provided so that the complainant can intuitively grasp the data.

Third, there is no guideline for service technology development through analyzing civil service data of government departments, public institutions and local governments. Therefore, it is possible to develop domestic technology products through the development of service technology for analysis of big data of public data and to present guidelines for utilization in public institutions.

In order to solve the present invention, the following problems are described and a solution is suggested to solve the service development approach.

First, it relies on the foreign data analysis platform, so it analyzes the existing open source and develops a platform suitable for the big data characteristic of the domestic public institution, and develops an analysis and visualization system suitable for the characteristics of the complaint data.

Second, there are real-time data processing limitations, various data processing limitations, and difficulty in managing a large number of distributed files. Therefore, we develop various distributed query processing techniques for real-time processing and develop batch processing module to support complex processing and analysis in big data processing and analysis.

Third, there is insufficient load distribution policy considering node workload and data utilization. Therefore, it is the development of data de-duplication and load balancing technology for efficient resource management.

According to the big data management system for the public service data service of the present invention, the local government can provide the policy decision, the public service statistics service and the public service prediction service through the real-time analysis of the civil service data of the public institutions, It has the effect of providing element technology and service for.

1 is a block diagram illustrating a big data management system for a public service data service of the present invention.
FIG. 2 is a block diagram illustrating a public complaint big data collector module of the present invention.
FIG. 3 is a block diagram illustrating a public private large data preprocessing and storage / distributed batch processing module of the present invention.
FIG. 4 is a flowchart illustrating a public private large data preprocessing module of the present invention.
FIG. 5 is a diagram for analyzing civil affairs statistics of the present invention. FIG.
FIG. 6 is a diagram for analyzing the classification of the present invention.
FIG. 7 is a schematic diagram of a keyword classification management and setting of the present invention.
FIG. 8 is a graph showing the state of civil complaints per month in a specific region of the present invention.
FIG. 9 is a diagram illustrating the analysis of complaint trends according to the top keywords of the present invention.

FIG. 1 and FIG. 2 illustrate public private data collecting module of the present invention, and collect data from a data collector module such as an Open API Source, a Blog Source, a News Source, and a Web Source. The data collector module will be described in detail as follows.

The Open API collects public application data from public agencies (data.go.kr) in real time. Web Crawler collects complaint data provided by public institution's website bulletin board through web crawler in real time. Web Scraper collects complaints data in real time on Web site bulletin boards of public institutions such as Web Crawler.

FIG. 3 is a block diagram of the public-private big data preprocessing and storage module. At this time, the Log Aggregator is a collection and management of complaint data logs collected from various collectors in FIG. 1 and FIG. The connector module passes the collected complaint data to the data preprocessor and filters the data required before the data preprocessing.

Figure 4 is a flowchart of the public complaint big data preprocessing module. The data preprocessor eliminates data redundancy that is collected from various collectors. In addition, tagging functions according to the data collection path are performed to distinguish real-time processing and batch processing according to collected data types and characteristics. Finally, the civil data, which is refined through the data preprocessor, is stored in the distributed file system (HDFS).

HDFS, HDFS server module, search engine index storage module, and DB storage module for storing real time data after data preprocessor is executed, real time storage management for storing DISK, HDFS, NoSQL, / RTI > Data stored in real time is distributed and processed through MapReduce, Hive, Pig, and Mahout modules.

The details of data mining, classification / grouping, subject analysis, and complaints analysis, which are necessary functions for data mining for public data analysis, are as follows.

Data mining provides in-memory based real-time complaint big data stream mining function by distributed processing technology and analysis mining technology for integrated mining of various types of complaint data provided from distributed multi-source. It also provides civil data mining, data identification, search, and context-based text mining.

The classification / grouping systematically classifies or groups the data and results obtained through the analysis of complainant big data such as similarity or ranking to provide efficient query processing and analysis and related services.

Theme analysis automatically analyzes the subject of public application data through context - based text mining analysis.

Analysis of complaints analysis analyzes public complaints trends and types by analyzing public complaints data and social network data. At this time, it is possible to analyze various kinds of civil affairs trends that are increasing through civil affairs types, subjects, and statistical analysis.

Details of the civil service, civil service, and civil service forecasting service, which are necessary functions for the public service, are as follows.

Civil affairs statistics service is a civil service inquiry and statistical service that provides related contents and related civil information search.

The civil service policy service is a service that supports public sector decision making and policy decision by analyzing contents and tendency of recent civil petitions.

The civil service prediction service is based on the contents of the civil affairs analysis, and establishes the public institution policy and the budget preparation service for the civil service which is expected in the future.

[Example 1]

For the sake of the present invention, Fig. 5 is a schematic diagram of civil service statistics. The total number of complaints on the upper right of Figure 5 is the total number of complaints collected from January 1, 2014 to November 20, 2014. It also shows numerical values and schematics by keyword ranking by region.

[Example 2]

For the sake of the present invention, FIG. 6 shows classification and management of a complaint keyword for a complaint classification service and a keyword status according to a region. FIG. 7 is a flowchart illustrating a method of classifying a complaint keyword according to a complaint keyword classification and setting management function, and inputting a complaint keyword related to a category. Finally, explain the category.

[Example 3]

For the sake of the present invention, FIG. 8 shows the state of civil affairs in Gyeonggi-do, which is a specific area, from January 2014 to November 2014. Figure 9 also shows the analysis and visualization of complaint trends for each top keyword.

Claims (7)

A data collector module for collecting public information;
A data connection module for transmitting the data collected by the data collector module to the data preprocessor;
A data preprocessor module for processing data received from the data connection module;
And a real-time public-private-data management module for real-time analysis and storage of the collected public-private large data.
The method according to claim 1,
The data collector module includes an Open API for collecting public cri- teria data provided by a public portal in real time;
A web crawler that collects civil petitions data provided by a public institution's web site bulletin board through a web crawler in real time;
And a log aggregator for collecting the complaint data logs from the various collectors.
The method according to claim 1,
Wherein the data preprocessor module unit comprises:
Removing data redundancy collected from various collectors;
A tagging function according to a data collection path for distinguishing between real-time processing and batch processing according to collected data types and characteristics;
And storing the civil complaint raw data refined through the data preprocessor in a distributed file system (HDFS).
The method of claim 2,
The data connection (connector)
Transmitting the collected complaint data from the Open API, the Web Crawler, and the Log Aggregator to the data preprocessor;
And filtering the data required before the data preprocessing.
The method according to claim 1,
The public private large data real-time storage management module includes:
A real-time processing module that performs in-memory caching and indexing functions for real-time processing, performs time unit and progressive analysis, and stores the time units and progressive analysis in the NoSQL DB;
An HDFS (Hadoop File System) module for storing civil information collected from various sources into a distributed file system;
The MapReduce module, which is a Map / Reduce framework module for complaint data processing for distributing and parallelizing a large amount of complicated big data in a cluster environment by mapping and distributing data with a distributed, parallel map and reduction algorithm in a large amount of distributed complainant data and;
An RDBMS module that stores large-volume complainant big data in a relational DB through Map / Reduce algorithm and provides standard distributed SQL processing functions by supporting standard SQL;
And a NoSQL (HBase) module, which is a module for storing and managing data in the NoSQL DB through Map / Reduce algorithm for large-volume civil complaint big data.
The method according to claim 1,
A classification and grouping module that provides efficient query processing and analysis related services by systematically classifying and grouping the data and results obtained through the analysis of similarity and rankings,
A subject analysis module for automatically analyzing the subject of public affairs data through context based text mining analysis;
It is a civil affairs trend analysis module that analyzes trends and types of public complaints through analysis of public complaint data and social network data,
And a data mining module for analyzing the public data of the civil information system.
The method according to claim 1,
A civil service statistical service module for searching and providing contents of the complaint request and the related complaint information;
A civil service policy service module for analyzing contents and tendencies of recent civil petitions and supporting public decision making and policy making;
Based on the contents of civil affairs analysis, it is a service module for civil service prediction,
A public data center, and a public data center.







KR1020140184749A 2014-12-19 2014-12-19 Big data management system for public complaints services KR20160075971A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
KR1020140184749A KR20160075971A (en) 2014-12-19 2014-12-19 Big data management system for public complaints services

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
KR1020140184749A KR20160075971A (en) 2014-12-19 2014-12-19 Big data management system for public complaints services

Publications (1)

Publication Number Publication Date
KR20160075971A true KR20160075971A (en) 2016-06-30

Family

ID=56352507

Family Applications (1)

Application Number Title Priority Date Filing Date
KR1020140184749A KR20160075971A (en) 2014-12-19 2014-12-19 Big data management system for public complaints services

Country Status (1)

Country Link
KR (1) KR20160075971A (en)

Cited By (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106951497A (en) * 2017-03-15 2017-07-14 深圳市德信软件有限公司 A kind of method and system based on Hadoop framework data analysis diagrammatic representation
KR101864509B1 (en) * 2016-11-30 2018-06-04 영남대학교 산학협력단 Method and system for analyzing bigdata
KR101877885B1 (en) * 2017-10-27 2018-07-12 주식회사 공간인소프트 information linking apparatus and method for public open data
WO2019088334A1 (en) * 2017-11-01 2019-05-09 (주)데이터스트림즈 System for storing and searching big data in real-time
KR20190062848A (en) 2017-11-29 2019-06-07 주식회사 비네아 System of big data mining using incremental learning and a method thereof
CN110111084A (en) * 2019-05-16 2019-08-09 上饶市中科院云计算中心大数据研究院 A kind of government affairs service hotline analysis method and system
KR102091529B1 (en) 2019-09-03 2020-03-23 (주)빅인사이트 Method and apparatus for training AI model using user's time series behavior data
KR102096328B1 (en) 2019-08-12 2020-04-02 최미숙 Platform for providing high value-added intelligent research information based on prescriptive analysis and a method thereof
KR102156287B1 (en) 2020-03-20 2020-09-15 주식회사 비네아 Platform for providing high value-added intelligent research information based on prescriptive analysis and a method thereof
KR102156289B1 (en) 2020-03-20 2020-09-15 주식회사 비네아 Curation system using platform of high value-added intelligent research information based on prescriptive analysis and a method thereof
KR20210028554A (en) 2020-03-13 2021-03-12 (주)빅인사이트 Method and apparatus for training AI model using user's time series behavior data
KR102249524B1 (en) * 2019-12-26 2021-05-11 한국국토정보공사 Apparatus and method for predicting civil complaints using data-based spatial analysis
KR20210063061A (en) 2019-11-22 2021-06-01 현대건설주식회사 after-services counting prediction system of an apartment houses and method thereof
KR102306932B1 (en) * 2020-11-10 2021-09-30 주식회사 토이코스 Method for responding crisis using civil complaint data and system thereof
KR102365391B1 (en) 2020-12-07 2022-02-21 조영찬 Labeling method of video data and donation method using the same
KR20220025632A (en) * 2020-08-24 2022-03-03 주식회사 긴트 Method and apparatus for filtering failure codes of agricultural machinery
KR20220143230A (en) * 2021-04-15 2022-10-25 동국대학교 산학협력단 Apparatus and method detecting malicious complaint
KR102464117B1 (en) * 2022-03-18 2022-11-07 에쓰오씨소프트 주식회사 Method and apparatus for analyzing and managing contents of public institution big data portal using artificial intelligence
CN116862455A (en) * 2023-09-01 2023-10-10 中国标准化研究院 Multi-mode-based government service complaint early warning method and device

Cited By (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101864509B1 (en) * 2016-11-30 2018-06-04 영남대학교 산학협력단 Method and system for analyzing bigdata
CN106951497A (en) * 2017-03-15 2017-07-14 深圳市德信软件有限公司 A kind of method and system based on Hadoop framework data analysis diagrammatic representation
KR101877885B1 (en) * 2017-10-27 2018-07-12 주식회사 공간인소프트 information linking apparatus and method for public open data
WO2019088334A1 (en) * 2017-11-01 2019-05-09 (주)데이터스트림즈 System for storing and searching big data in real-time
KR20190062848A (en) 2017-11-29 2019-06-07 주식회사 비네아 System of big data mining using incremental learning and a method thereof
CN110111084A (en) * 2019-05-16 2019-08-09 上饶市中科院云计算中心大数据研究院 A kind of government affairs service hotline analysis method and system
KR102096328B1 (en) 2019-08-12 2020-04-02 최미숙 Platform for providing high value-added intelligent research information based on prescriptive analysis and a method thereof
KR102091529B1 (en) 2019-09-03 2020-03-23 (주)빅인사이트 Method and apparatus for training AI model using user's time series behavior data
KR20210063061A (en) 2019-11-22 2021-06-01 현대건설주식회사 after-services counting prediction system of an apartment houses and method thereof
KR102249524B1 (en) * 2019-12-26 2021-05-11 한국국토정보공사 Apparatus and method for predicting civil complaints using data-based spatial analysis
KR20210028554A (en) 2020-03-13 2021-03-12 (주)빅인사이트 Method and apparatus for training AI model using user's time series behavior data
KR102156289B1 (en) 2020-03-20 2020-09-15 주식회사 비네아 Curation system using platform of high value-added intelligent research information based on prescriptive analysis and a method thereof
KR102156287B1 (en) 2020-03-20 2020-09-15 주식회사 비네아 Platform for providing high value-added intelligent research information based on prescriptive analysis and a method thereof
KR20220025632A (en) * 2020-08-24 2022-03-03 주식회사 긴트 Method and apparatus for filtering failure codes of agricultural machinery
WO2022108052A1 (en) * 2020-08-24 2022-05-27 주식회사 긴트 Method and apparatus for filtering fault codes of agricultural machine
KR102306932B1 (en) * 2020-11-10 2021-09-30 주식회사 토이코스 Method for responding crisis using civil complaint data and system thereof
KR102365391B1 (en) 2020-12-07 2022-02-21 조영찬 Labeling method of video data and donation method using the same
KR20220143230A (en) * 2021-04-15 2022-10-25 동국대학교 산학협력단 Apparatus and method detecting malicious complaint
KR102464117B1 (en) * 2022-03-18 2022-11-07 에쓰오씨소프트 주식회사 Method and apparatus for analyzing and managing contents of public institution big data portal using artificial intelligence
CN116862455A (en) * 2023-09-01 2023-10-10 中国标准化研究院 Multi-mode-based government service complaint early warning method and device

Similar Documents

Publication Publication Date Title
KR20160075971A (en) Big data management system for public complaints services
US11582123B2 (en) Distribution of data packets with non-linear delay
US11449562B2 (en) Enterprise data processing
Sebei et al. Review of social media analytics process and big data pipeline
US10599697B2 (en) Automatic topic discovery in streams of unstructured data
US20170168751A1 (en) Optimization for Real-Time, Parallel Execution of Models for Extracting High-Value Information from Data Streams
CN104951512A (en) Public sentiment data collection method and system based on Internet
US10698935B2 (en) Optimization for real-time, parallel execution of models for extracting high-value information from data streams
Bellini et al. Data flow management and visual analytic for big data smart city/IOT
US8484217B1 (en) Knowledge discovery appliance
US10127617B2 (en) System for analyzing social media data and method of analyzing social media data using the same
KR101532252B1 (en) The system for collecting and analyzing of information of social network
US20120030164A1 (en) Method and system for gathering and usage of live search trends
KR101665649B1 (en) System for analyzing social media data and method for analyzing social media data using the same
Wadhera et al. A systematic Review of Big data tools and application for developments
Mavrogiorgos et al. Self-Adaptable Infrastructure Management for Analyzing the Efficiency of Big Data Stores
KR20210045172A (en) Big Data Management and System for Livestock Disease Outbreak Analysis
CN116467291A (en) Knowledge graph storage and search method and system
Martínez-Castaño et al. Polypus: a big data self-deployable architecture for microblogging text extraction and real-time sentiment analysis
KR101718599B1 (en) System for analyzing social media data and method for analyzing social media data using the same
Shouaib et al. Survey on iot-based big data analytics
Xu et al. The application of web crawler in city image research
KR20210037488A (en) Big Data Analytics-Based Advertising Marketing System
CN113505172B (en) Data processing method, device, electronic equipment and readable storage medium
Gao et al. Unified Searching Service for Electric Big Data

Legal Events

Date Code Title Description
A201 Request for examination
E902 Notification of reason for refusal
E601 Decision to refuse application