KR101600275B1 - Real-Time Big Data Event Processing System - Google Patents

Real-Time Big Data Event Processing System Download PDF

Info

Publication number
KR101600275B1
KR101600275B1 KR1020150038710A KR20150038710A KR101600275B1 KR 101600275 B1 KR101600275 B1 KR 101600275B1 KR 1020150038710 A KR1020150038710 A KR 1020150038710A KR 20150038710 A KR20150038710 A KR 20150038710A KR 101600275 B1 KR101600275 B1 KR 101600275B1
Authority
KR
South Korea
Prior art keywords
big data
real
real time
time
module
Prior art date
Application number
KR1020150038710A
Other languages
Korean (ko)
Inventor
서영우
Original Assignee
주식회사 투그램시스템즈
서영우
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 주식회사 투그램시스템즈, 서영우 filed Critical 주식회사 투그램시스템즈
Priority to KR1020150038710A priority Critical patent/KR101600275B1/en
Application granted granted Critical
Publication of KR101600275B1 publication Critical patent/KR101600275B1/en

Links

Images

Classifications

    • G06F17/30318—
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/30595—

Abstract

The present invention relates to a real-time big data event processing system. More specifically, by implementing a process of processing big data events such as collection / storage / analysis / reporting in real time in order to smoothly transfer data between a specific big data and a user providing a service, It is convenient and easy to utilize the complicated and semi-formal big data which is formal and semi-formalized by the user, and manage big data transmitted in non-real time and big data transmitted in real time as a separate process , And non-real-time big data is required to be compared with real-time big data, it is transmitted to the real-time analyzer, and the non-real time big data that is firstly analyzed is not required to be duplicated, Real-time Big Data to Improve Processing Speed It relates to a tree processing machine.

Description

Real time Big Data Event Processing System {omitted}

The present invention relates to a real-time big data event processing system. More specifically, by implementing a process of processing big data events such as collection / storage / analysis / reporting in real time in order to smoothly transfer data between a specific big data and a user providing a service, It is convenient and easy to utilize the complicated and semi-formal big data which is formal and semi-formalized by the user, and manage big data transmitted in non-real time and big data transmitted in real time as a separate process , And non-real-time big data is required to be compared with real-time big data, it is transmitted to the real-time analyzer, and the non-real time big data that is firstly analyzed is not required to be duplicated, Real-time Big Data to Improve Processing Speed It relates to a tree processing machine.

Big data refers to a large amount of data, which is digital data generated through a web log or a social network of a user information database recorded in a web server.

In order to process such a large amount of data, a technology capable of collectively processing big data is needed. In recent years, research and development of big data processing has been actively carried out.

Korean Patent Registration No. 1463974 discloses a prior art document on a big data analysis system and method for marketing.

The prior art document searches information according to a search criterion set from a predetermined type of website, parses the search information, and converts the search information into a format based on a hierarchical common code and an item, A distributed storage unit for distributedly storing the information converted by the information collecting unit with reference to the common code and the size, and a distributed storage unit for accessing distributed information stored in a distributed processing manner according to a specific analysis process, An analyzing unit for storing the result of analysis in the utilization database and re-analyzing the results of the analysis of the new value and storing the result in the utilization database; And then output the result, and output And provides the data to the distributed storage unit according to the data format or stores it in the utilization database for reuse.

However, the big data analysis processing method disclosed in the existing prior art documents is a method in which a single collection section manages regular, semi-structured, and non-orthogonal large data generated from the web or conventional data in a collective manner, There is a problem in that the processing speed of the process is slowed down and the efficiency is low.

SUMMARY OF THE INVENTION The present invention has been made in order to solve the above-mentioned problems, and it is an object of the present invention to provide a method and apparatus for processing large data, which are unstructured, have.

In order to achieve the above object, a real-time big data event processing system according to the present invention, which is devised to achieve the above object, collects arbitrary big data incoming from a public entity, wherein the big data includes big data transmitted in non- A collecting module for performing an event function separately collecting and designating as an item including a protocol and a type of big data; It analyzes the big data collected by the collection module to find out what type of big data, analyzes whether there is correlation between the big data, maps it to the form required by the user, and transmits the analyzed big data to the relational database. An analysis module including an analyzer; A generation module for performing an event function for transforming and processing the analysis result of the analysis module into a format and a protocol desired by the user before delivering the analysis result to the user; A notification module that performs an event function of notifying a part of the big data transmitted in non-real time among the big data to the real time big data analyzer; And the analyzed big data may be stored in a relational database and converted into a form including a dashboard and a chart if necessary so that the user can visually recognize the data and utilize it in business.

In addition, the analysis module can separately analyze the big data transmitted in the non-real time and the big data transmitted in real time, and then, the analyzed data can be stored in the relational database.

In addition, the notification module selectively receives the big data determined to be related to the big data transmitted in real time among the big data transmitted in the non-real time accumulated in the collection module, notifies the real big data analyzer, and the real time big data analyzer It is possible to transmit the analyzed big data to the relational database by comparing and analyzing the associations.

According to the present invention, by implementing a process of processing a big data event such as collection / storage / analysis / report in real time in order to smoothly transfer data between a certain big data provider and a user utilizing the service, It has the effect of processing the complicated and massive big data of a formal and semi-formal type to a user (a financial institution, a manufacturing organization, etc.) in a standardized form so that the user can conveniently and easily utilize it.

According to the present invention, the big data to be transmitted in real time and the big data to be transmitted in real time are managed by a separate process. When comparative analysis with real time big data is required among non-real time big data, Therefore, it is unnecessary to duplicate and analyze non-real-time big data that has been subjected to the first-order analysis, thereby improving the processing speed.

1 is a conceptual diagram of a real-time big data event processing system according to a preferred embodiment of the present invention,
2 is a configuration diagram of a real-time big data event processing system according to a preferred embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same reference numerals are used to designate the same or similar components throughout the drawings. In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear. In addition, the preferred embodiments of the present invention will be described below, but it is needless to say that the technical idea of the present invention is not limited thereto and can be variously modified by those skilled in the art.

FIG. 1 is a conceptual diagram of a real-time big data event processing system according to a preferred embodiment of the present invention, and FIG. 2 is a configuration diagram of a real-time big data event processing system according to a preferred embodiment of the present invention.

The real-time big data event processing system according to the preferred embodiment of the present invention provides a big data event processing service based on web and mobile and includes a collection module 100, an analysis module 200, a generation module 300, 400, and a reporting module 500.

The collection module 100 collects any big data coming from the public entity. Big data flowing into the acquisition module 100 are collected separately as big data to be transmitted in non-real time and large data to be transmitted in real time, and the analysis process also proceeds individually.

The collection module 100 specifies the protocol of the collected big data and the type of the big data, and performs an event to specify and map the stream to be used in the analysis of the big data.

The collection module 100 may apply a NoSQL (Not Only SQL) method in which the collected big data is divided and allocated to a plurality of servers.

The analysis module 200 separates and analyzes the big data transmitted in non-real time and the big data transmitted in real time. The big data transmitted in real time is analyzed by real time big data analyzer. Big data that is related to big data transmitted in real time among non-real time big data can be compared and analyzed with real time big data analyzer.

In the analysis, the analysis is performed using an open source real-time big data analyzer and a batch layer, and the analyzed data is transferred to a relational database in real time by mapping the data to a reportable form to the user.

The analysis module 200 collects the big data and automatically analyzes the correlation data during analysis and processes the data in real time. The analysis module 200 applies and analyzes at least one of CEP (Complex Event Processing) and hive-based analyzers, (Complex Event Processing) can be analyzed in a time-wise manner through a hive-based analyzer.

The analysis module 200 enables quick analysis using a high-performance real-time big data analyzer. Here, the high-performance real-time Big Data Analyzer means that big data of 2.5M or more can be processed.

The analysis module 200 provides a scalable query language for real-time big data analysis, filtering according to conditions, and enabling time series processing.

The generation module 300 converts the analysis result of the analysis module 200 into a format and a protocol desired by the user, and processes the result so as to provide it to the user. At this time, the analysis results are stored in the relational database in real time.

The generation module 300 supports an event function such as notification to the user through the form of E-mail or SMS.

The notification module 400 notifies the real time big data analyzer of a part of the big data transmitted in non-real time among the big data.

More specifically, the notification module 400 selectively receives big data related to the big data transmitted in real time among the large data accumulated in the acquisition module 100 in real time, And the real time Big Data Analyzer compares and analyzes the correlations and transmits the analyzed big data to the relational database.

The reporting module 500 receives large data analyzed by the analysis module 200 and converts the data into charts and dashboards so that actual users (financial institutions, manufacturers, medical institutions, general citizens, etc.) And displays it.

Finally, the user does not simply receive the raw data, which is not processed, but processes it through the real-time big data event processing system of the present invention and receives it in a formalized form so that it can be directly applied to the business.

The real-time big data event processing system according to the preferred embodiment of the present invention classifies random big data related to various fields such as traffic information, airport and port-based logistics information, application performance, and log information through a real time big data analyzer, And provides the information to the users who need it so that the user can utilize it efficiently and reasonably.

It will be apparent to those skilled in the art that various modifications, substitutions and substitutions are possible, without departing from the scope and spirit of the invention as disclosed in the accompanying claims. will be. Therefore, the embodiments disclosed in the present invention and the accompanying drawings are intended to illustrate and not to limit the technical spirit of the present invention, and the scope of the technical idea of the present invention is not limited by these embodiments and the accompanying drawings . The scope of protection of the present invention should be construed according to the following claims, and all technical ideas within the scope of equivalents should be construed as falling within the scope of the present invention.

100 - Collection module 200 - Analysis module
300 - Generation module 400 - Notification module
500 - Reporting module

Claims (3)

Collects any big data coming from a public institution, and big data collects into big data transmitted in non-real time and big data transmitted in real time, and designates as an item including protocol and type of big data A collecting module for performing the collecting process;
It analyzes the big data collected by the collection module to find out what type of big data, analyzes whether there is correlation between the big data, maps it to the form required by the user, and transmits the analyzed big data to the relational database. An analysis module including an analyzer;
A generation module for performing an event function for transforming and processing the analysis result of the analysis module into a format and a protocol desired by the user before delivering the analysis result to the user;
A notification module that performs an event function of notifying a part of the big data transmitted in non-real time among the big data to the real time big data analyzer; And
Once the analysis is completed, the big data is stored in a relational database and converted into a form including a dashboard and a chart, if necessary, so that the user can visually recognize the data and utilize it in business
Time event data. The method according to claim 1,
Analysis module is a real-time big data event processing system that separately analyzes big data transmitted in non-real time and big data transmitted in real time, and merges and stores them in relational database. The method according to claim 1,
The notification module selectively receives the big data that is determined to be related to the big data transmitted in real time among the big data transmitted in the non-real time accumulated in the collection module, notifies the real big data analyzer, and the real time big data analyzer notifies the correlation A real-time big data event processing system that transmits the analyzed big data to a relational database.





KR1020150038710A 2015-03-20 2015-03-20 Real-Time Big Data Event Processing System KR101600275B1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
KR1020150038710A KR101600275B1 (en) 2015-03-20 2015-03-20 Real-Time Big Data Event Processing System

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
KR1020150038710A KR101600275B1 (en) 2015-03-20 2015-03-20 Real-Time Big Data Event Processing System

Publications (1)

Publication Number Publication Date
KR101600275B1 true KR101600275B1 (en) 2016-03-07

Family

ID=55540335

Family Applications (1)

Application Number Title Priority Date Filing Date
KR1020150038710A KR101600275B1 (en) 2015-03-20 2015-03-20 Real-Time Big Data Event Processing System

Country Status (1)

Country Link
KR (1) KR101600275B1 (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101906357B1 (en) * 2017-09-26 2018-10-10 한국건설기술연구원 Collecting Device and Method of Various Big Data
KR20180126792A (en) * 2017-05-18 2018-11-28 주식회사 알티베이스 System and Method for processing complex stream data using distributed in-memory
KR102569704B1 (en) * 2023-04-07 2023-08-25 (주)디에스티인터내셔널 Apparatus and method for activating user-customized heterogeneous big data real-time dashboard

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101503701B1 (en) * 2014-04-30 2015-03-20 (주)아이비즈소프트웨어 Method and Apparatus for Protecting Information Based on Big Data

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101503701B1 (en) * 2014-04-30 2015-03-20 (주)아이비즈소프트웨어 Method and Apparatus for Protecting Information Based on Big Data

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20180126792A (en) * 2017-05-18 2018-11-28 주식회사 알티베이스 System and Method for processing complex stream data using distributed in-memory
KR101982756B1 (en) * 2017-05-18 2019-05-28 주식회사 알티베이스 System and Method for processing complex stream data using distributed in-memory
KR101906357B1 (en) * 2017-09-26 2018-10-10 한국건설기술연구원 Collecting Device and Method of Various Big Data
KR102569704B1 (en) * 2023-04-07 2023-08-25 (주)디에스티인터내셔널 Apparatus and method for activating user-customized heterogeneous big data real-time dashboard

Similar Documents

Publication Publication Date Title
CN108509326B (en) Service state statistical method and system based on nginx log
CN105095211B (en) The acquisition methods and device of multi-medium data
CN104899314B (en) A kind of parentage analysis method and apparatus of data warehouse
KR20160075971A (en) Big data management system for public complaints services
CN111967761B (en) Knowledge graph-based monitoring and early warning method and device and electronic equipment
Nasridinov et al. A decision tree-based classification model for crime prediction
CN103942210A (en) Processing method, device and system of mass log information
CN106383887A (en) Environment-friendly news data acquisition and recommendation display method and system
US10706062B2 (en) Method and system for exchanging data from a big data source to a big data target corresponding to components of the big data source
WO2013185601A1 (en) Method and device for obtaining product information and computer storage medium
WO2013106595A2 (en) Processing store visiting data
CN104182465A (en) Network-based big data processing method
KR101600275B1 (en) Real-Time Big Data Event Processing System
CN104572976A (en) Website data updating method and system
CN111913860B (en) Operation behavior analysis method and device
Li et al. City digital pulse: a cloud based heterogeneous data analysis platform
Dobos et al. A multi-terabyte relational database for geo-tagged social network data
KR102361112B1 (en) Extracting similar group elements
CN112965979A (en) User behavior analysis method and device and electronic equipment
US8589404B1 (en) Semantic data integration
CN106557483B (en) Data processing method, data query method, data processing equipment and data query equipment
CN106202509A (en) A kind of processing method of log information
KR102345410B1 (en) Big data intelligent collecting method and device
CN117251414A (en) Data storage and processing method based on heterogeneous technology
CN107729206A (en) Real-time analysis method, system and the computer-processing equipment of alarm log

Legal Events

Date Code Title Description
E701 Decision to grant or registration of patent right
GRNT Written decision to grant
FPAY Annual fee payment

Payment date: 20190318

Year of fee payment: 4