KR20210045172A - Big Data Management and System for Livestock Disease Outbreak Analysis - Google Patents

Abstract

While big data has emerged as a global concern for information technology, attention is focused on what a public institution and companies will create which value through the big data collected so far. Therefore, a big data management system for a livestock disease outbreak information service constitutes development stages as follows. Firstly, the present invention develops a big data collector module for livestock disease outbreak information to collect disease outbreak data provided from public/private institutions as various sources in web portals, an SNS, and intranet inside public institutions in real time. Secondly, the present invention develops a big data real-time storing manager module for livestock disease outbreak information to store and manage the collected big data through a Hadoop distributed file system (HDFS), and store and manage the big data in a relational DB by performing real-time distributed parallel processing through a map reduce framework. Thirdly, the present invention develops a big data analyzing and visualizing processor module to analyze data for each subject by using real-time data mining technology from the stored livestock disease outbreak data and predict classification, grouping, and disease outbreak progress. Finally, the present invention develops a livestock disease outbreak information analyzing services system to effectively provide a service for disease outbreak statistics, disease outbreak policies, and disease outbreak prediction for strengthen national ability to respond to the livestock disease outbreak from the analyzed livestock disease outbreak information data.

Description

Big Data Management and System for Livestock Disease Outbreak Analysis}

The present invention is a technology for prediction and visualization services through data analysis as well as data collection, storage, processing, and management technologies in order to utilize big data of public/private organizations.

Big data is rapidly emerging as an interest in information technology around the world, and attention is focused on what value public institutions and companies will create through big data collected so far. Hadoop, an open source project for big data processing analysis, is a Hadoop file system (HDFS), OS level abstractions, and MapReduce engine, which can easily aggregate, query, and analyze large amounts of data. It includes HIVE, Mahout, a machine learning library capable of distributed and parallel processing for the development of intelligent applications that require large amounts of data. It also consists of necessary Java archive files (Java ARchive, JAR), script to start Hadoop, source codes, and related materials.

In the case of big data of livestock disease occurrence information, the amount of data smaller than the size of the data is divided into various departments, and data are not linked to each other and are not separately managed and used. Therefore, various approaches and analysis methods are needed for big data of livestock disease occurrence information. An object of the present invention is to provide a livestock disease occurrence information big data collector, livestock disease occurrence information big data storage manager, livestock disease occurrence information big data analysis and visualization processor, and livestock disease occurrence information statistics and prediction services.

In order to solve the present invention with a technical development approach, the problem is described as follows and a solution is proposed.

First, disease data are divided and managed by public institutions, local governments, government departments, and private institutions, so the data is not linked. Therefore, it provides multi-source big data collection, storage management, analysis, and visualization services that consider various sources and formats for livestock disease occurrence information data.

Second, a service using some livestock disease outbreak information data has been developed by government agencies, but there is no case in which disease outbreak information service has been commercialized. Therefore, it provides convenience so that consumers can intuitively grasp data by providing various visualization functions such as charts and graphs.

Third, there is no guideline for service technology development through analysis of information on livestock disease occurrence information of government ministries, public institutions, and local governments. Therefore, through the development of service technology for big data analysis of public data, it is possible to develop domestic technology products and present guidelines that can be used by public institutions.

In addition, in order to solve the present invention with a service development approach, the following problems are described and a solution is proposed.

First, since it is dependent on a foreign big data analysis platform, it is to analyze existing open sources to develop a platform suitable for the characteristics of big data in domestic public institutions, and to develop an analysis and visualization system suitable for the characteristics of disease occurrence data.

Second, there are limitations in real-time data processing, limitations in processing various data, and difficulties in managing a large number of distributed files. Therefore, it is a batch processing module to develop various distributed query processing techniques for real-time processing and to support complex and various processing and analysis in big data processing and analysis.

Third, the load balancing policy considering the workload and data utilization of the node is insufficient. Therefore, it is the development of data deduplication and load balancing technology for efficient resource management.

According to the present invention, through real-time analysis of livestock disease occurrence information data of public/private organizations, local governments can provide policy decisions, livestock disease occurrence information statistics service, and livestock disease occurrence information prediction service. In addition, it is the development of elemental technologies and services to provide customized livestock disease occurrence information services.

1 is a flow chart of a big data collector module for livestock disease occurrence information according to the present invention.
Figure 2 is a flow chart of the livestock disease occurrence information big data pre-processing and storage/distributed batch processing module of the present invention.
3 is a flow chart of a big data preprocessing module for livestock disease occurrence information according to the present invention.
Figure 4 is a schematic diagram of the livestock disease occurrence information statistical service of the present invention.
5 is a schematic diagram of a livestock disease occurrence information classification service according to the present invention.
6 is a diagram illustrating management and setting of disease keyword classification according to the present invention.
7 is a service for decorating the current status of disease occurrence by month in a specific area according to the present invention.
8 is a disease trend analysis and visualization service for each top keyword of the present invention.

1 is a livestock disease occurrence information big data collector module of the present invention, and livestock disease occurrence data (Open API Source, Blog Source, News Source, Web Source, etc.) are collected by the data collector module. A detailed description of the data collector module is as follows. Open API collects livestock disease occurrence information data of public institutions provided by the public portal (data.go.kr) in real time. The Web Crawler collects disease occurrence data provided on the disease occurrence report screens of domestic and foreign public institutions in real time through a web crawler. Web Scraper, like Web Crawler, collects disease occurrence information data in real time from websites of domestic and foreign public institutions.

2 is a module for preprocessing and storing big data of livestock disease occurrence information. At this time, the Log Aggregator is a disease occurrence data log collection and management collected by various collectors in FIG. 1. The connector module transfers the disease occurrence data collected by the collector to the data preprocessor, and filters the necessary data before data preprocessing.

3 is a flow chart of a big data preprocessing module for livestock disease occurrence information. The data preprocessor eliminates duplication of data collected from various collectors. In addition, in order to classify real-time processing and batch processing according to the collected data type and characteristics, it performs a tagging function according to the data collection path. Finally, the raw data of disease occurrence, purified through the data preprocessor, is stored in the distributed file system (HDFS).

After executing the data preprocessor, a disk server module, HDFS server module, search engine index storage module, and DB storage module are temporarily placed for real-time data storage to store DISK, HDFS, NoSQL, search engine, and database respectively. Is stored in. Data stored in real time is distributed and batch processed through MapReduce, Hive, Pig, and Mahout modules.

Details of data mining, classification/grouping, topic analysis, and disease outbreak trend analysis, which are necessary functions for data mining for big data analysis of livestock disease occurrence information, are as follows.

Data mining is a distributed processing technology and analysis mining technology for integrated mining of various types of disease outbreak data provided from distributed multiple sources, and provides in-memory-based real-time disease outbreak big data stream mining. In addition, it provides the function of extracting, identifying, and searching for disease occurrence big data patterns, and the context-based text mining function.

Classification/grouping provides efficient query processing and analysis and related services by systematically classifying or grouping data and results obtained through analysis of big data on disease occurrence such as similarity or ranking.

Thematic analysis automatically analyzes the subject of livestock disease occurrence information data through context-based text mining analysis.

Disease occurrence trend analysis analyzes domestic and foreign disease occurrence trends and types through analysis of livestock disease occurrence information data and social network data. At this time, it is possible to analyze the trend of various types of disease occurrence through the analysis of disease occurrence types, topics, and statistics.

Details of the disease occurrence statistics service, disease occurrence policy service, and disease occurrence prediction service, which are necessary functions for processing livestock disease occurrence information versus service, are as follows.

The disease occurrence statistics service is a disease occurrence inquiry and statistics service that provides a related search for disease occurrence request details and related disease occurrence information.

The disease outbreak policy service is a service that supports decision-making and policy-making by public institutions by analyzing recent disease outbreaks and trends.

Disease outbreak prediction services include public institution policy establishment and budget preparation services for future disease outbreak services that are predicted based on the analysis of disease outbreaks.

[Example 1]

4 is a schematic diagram of a disease occurrence statistics service for the effectiveness of the present invention. The total number of disease outbreaks at the top of FIG. 4 is the total number of disease outbreaks collected from January 1, 2019 to October 15, 2019. And it shows figures and diagrams by keyword ranking by country.

[Example 2]

For the effectiveness of the present invention, FIG. 5 shows the classification and management of disease occurrence keywords for the disease occurrence classification service and the current status of keywords by region. 6 is a disease occurrence keyword classification and setting management function, in order to classify disease occurrence keywords, a category is created and a disease occurrence keyword related to the category is input. Finally, the categories are explained.

[Example 3]

For the practicality of the present invention, FIG. 7 shows the status of monthly disease occurrence in Gyeonggi-do, which is a specific region, from January 2019 to October 2019. In addition, FIG. 8 shows the analysis and visualization of disease occurrence trends by top keywords.

Claims (18)

It is a data collector module, a connection part, and a data preprocessor module part for collecting big data of livestock disease occurrence information. Based on the module unit, it is divided into a module that collects livestock disease occurrence information data, a module that transfers the collected data to a data preprocessor, and a data preprocessor module unit. The method of claim 1, wherein the detailed modules of the data collector module are Open API, Web Crawler, and Log Aggregator. The Open API module collects livestock disease occurrence information data of public institutions provided by public portals in real time. The Web Crawler module collects disease occurrence data provided by public institutions' website bulletin boards in real time through a web crawler. The Log Aggregator module collects disease occurrence data logs from various collectors. The method of claim 1, wherein the data preprocessor module removes redundancy of data collected from various collectors. In addition, it performs a tagging function according to the data collection path to classify it into real-time processing and batch processing according to the collected data type and characteristics. Finally, the raw data of disease occurrence, purified through the data preprocessor, is stored in the distributed file system (HDFS). The method according to claim 1, wherein the data connector module transmits disease occurrence data collected from an Open API, a Web Crawler, and a Log Aggregator collector to a data preprocessor, and filters necessary data before data preprocessing. The livestock disease occurrence information big data storage management module unit analyzes and stores livestock disease occurrence information big data collected in claim 1 in real time. Detailed modules include real-time processing module, HDFS (Hadoop File System), MapReduce, RDBMS, and NoSQL (HBase) module. The method of claim 5, wherein the real-time processing module performs in-memory caching and indexing functions for real-time processing. Then, it performs hourly or gradual analysis and stores it in NoSQL DB or RDBMS. The method of claim 5, wherein the HDFS module stores disease occurrence big data collected from various sources in a distributed file system. The method of claim 5, wherein the MapReduce module divides and stores a large amount of distributed disease occurrence data by mapping the data using a distributed/parallel Map/Reduce algorithm. It is also a Map/Reduce framework module for processing disease-occurrence data for distributed/parallel processing of large-capacity disease-occurring big data in a cluster environment. The method of claim 5, wherein the RDBMS module is a data storage that stores large-scale disease occurrence big data in a relational DB through a Map/Reduce algorithm. This module supports standard SQL and provides necessary distributed query processing functions. The method of claim 5, wherein the NoSQL (HBase) module is a module for storing and managing large-scale disease occurrence big data in a NoSQL DB through a Map/Reduce algorithm. The data mining module for analyzing livestock disease occurrence information big data includes classification/grouping, subject analysis, and disease occurrence trend analysis modules. The method of claim 11, wherein the classification/grouping module is a module for providing efficient query processing and analysis-related services by systematically classifying or grouping data and results obtained through analysis of big data on disease occurrence such as similarity or ranking. The method of claim 11, wherein the subject analysis module is a module that automatically analyzes the subject of livestock disease occurrence information data through context-based text mining analysis. The method of claim 11, wherein the disease occurrence trend analysis module is a module that analyzes domestic and foreign disease occurrence trends and types through analysis of livestock disease occurrence information data and social network data. The livestock disease outbreak information public service processing module is a disease outbreak statistics service, disease outbreak policy service, and disease outbreak prediction service module. The method of claim 15, wherein the disease occurrence statistics service module is a disease occurrence inquiry and statistics service module that provides a related search and provides information on a disease occurrence request and related disease occurrence information. The method of claim 15, wherein the disease outbreak policy service module is a service module that supports decision-making and policy-making of public institutions by analyzing the content and trend of recent disease outbreaks. The method of claim 15, wherein the disease occurrence prediction service module is a service module for establishing a public institution's policy and budget preparation for a disease occurrence service predicted in the future based on the analysis of disease occurrence.

Images