KR20150081126A - Big data service system based on web server and big data cluster using API driver - Google Patents

Abstract

The present invention relates to a method of data communication between a web server and a big data cluster using an API driver. The method includes: a step of converting a sensed signal into data in a java script object notation (JSON) code and transmits the data to the web server by a sensor; and a step of parsing the converted data in the JSON code transmitted to the web server by using a parser library, map-reducing the converted data, and storing the converted data in the big data cluster by an API driver.

Description

[0001] The present invention relates to a large data service system based on a web server and a large data cluster using an API driver,

The present invention relates to a large data service system, and more particularly, to a large data service system that transmits data of a JSON (JavaScript object notation) code transmitted from a sensor to a web server, parses data of a JSON code from a web server, (application programming interface) driver that analyzes the big data and outputs it to the web page when the user logs in to the web page and requests the big data, ≪ / RTI >

Search on the Web is essential to find out what you want quickly and efficiently in a heap of data that has a large amount of unstructured data. Web search engine developers are making various attempts to solve these problems. With the proliferation of the digital economy, big data environments are emerging around us, where large amounts of information and data can be produced.

Big Data is an existing database management system that refers to a large set of formal or unstructured data beyond the level to be collected, stored, managed and analyzed, and to extract values from these data and analyze the results. Big data processing has become an issue in terms of size, speed and diversity. Here, the volume refers to the size of data to be collected, stored, and analyzed, the velocity refers to a speed required for storing and processing data, the variety means a type of data, It is used to mean not only standardized data but also unstructured data such as image, voice, and tweeter.

However, in the conventional big data-related platform, there is no communication method between the web server and the big data cluster, which makes it difficult for the user to easily use the big data.

Patent Document 1 does not disclose a solution to a big data service system for a driver using a cloud service or a data communication method between a web server and a big data cluster.

1. Korean Registered Patent No. 10-1306891 (registered on September 04, 2013)

In order to solve the above problems, the present invention provides a method and system for communicating data between a web server and a big data cluster by using an API driver that can be applied to all situations without changing a program, We want to provide a big data analysis system that shows the data to the web platform.

A method for data communication between a web server and a big data cluster using an API driver according to the present invention for solving the above problem comprises converting a signal detected by a sensor into data of a JSON (javascript object notation) code and transmitting the data to a web server And parsing the data converted into JSON code transmitted from the application programming interface (API) driver to the web server using a parser library and mapreduce the data and store the parsed data in a big data cluster .

As another embodiment, a big data service system using an API driver according to the present invention includes a sensor for converting a sensed signal into data of a JSON code and transmitting the converted data to a web server, a web receiving a big data analysis request signal through a login of the user, A server, an API driver for parsing the data transmitted to the web server in a predetermined unit using a parser library, de-parsing the parsed unit data, arranging the parsed unit data in a predetermined order, and storing the parsed unit data in a big data cluster, And a big data cluster distributedly stored in a storage of the storage medium.

In another embodiment, the personalized health care service system using Big Data according to the present invention detects at least one of heart rate, body temperature, body mass index and blood sugar, converts the data into JSON code data, An API driver which receives data of a JSON code transmitted from the biosensor, parses the received JSON code from the web server, and parses and maps the data to a big data cluster, and a big data cluster And a control unit.

According to another embodiment of the present invention, there is provided a traffic information providing system using Big Data according to the present invention, comprising: a navigation system having a built-in GPS sensor to transmit navigation path or location information of a vehicle to a web server; And an API driver for parsing and mapping and a big data cluster for distributing and storing the parsed data transmitted from the API driver.

The present invention can be widely applied to most big data analysis systems related to data input / output processing between a web server and a big data cluster rather than a specific application.

The present invention can build a big data service system capable of collecting, storing, managing and analyzing big data suitable for all environments of users.

The present invention can reduce the construction cost of a big data service system because the user can utilize the big data stored in the big data cluster by logging in through a web page such as a blog or a facebook and the public can easily and simply use the big data service system Can be utilized.

INDUSTRIAL APPLICABILITY The present invention can provide a personalized big data analysis service optimized for each user by utilizing individual information of users transmitted from the sensor as big data.

1 is a schematic flowchart of a data input method between a web server and a big data cluster using an API driver according to the present invention.
2 is a schematic flowchart of a data output method between a web server and a big data cluster using an API driver according to the present invention.
3 is a schematic configuration diagram of a big data service system according to the present invention;
4 is a detailed configuration diagram of the API driver shown in FIG.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. The drawings illustrate only the essential features of the invention in order to facilitate clarity of the invention and are not to be construed in a limiting sense since they are not shown in the accompanying drawings.

The present invention transmits data, which is an aggregate of user-related information of JSON codes transmitted from the sensor 1, to the web server 2, and the API driver 3 parses the big data into unit data, And the data is distributed and stored in the big data cluster 4. The user then analyzes the relevant big data from the big data cluster 4 in order to provide the big data information requested through the web page, To a web platform to store, manage, and analyze massive data exceeding the size of a conventional database management system (DBMS) in an accurate and error-free manner.

Also, unlike the existing big data system, when the user logs in through a web page such as a blog or a facebook, the information requested by the user can be easily extracted using the big data, so that the user convenience is greatly increased. The profit generating effect can be largely obtained.

Hereinafter, a data communication method between a web server and a big data cluster using the API driver 3 according to the present invention and a big data service system using the method will be described with reference to FIG. 1 to FIG.

1 is a schematic flowchart of a data input method between a web server and a big data cluster using an API driver according to the present invention.

The sensor 1 periodically transmits the sensed signal to a web page URL (uniform resource locator) of the web server 2. The information collected by the sensor 1 may vary depending on the application of the big data analysis system, and may be, for example, position information, temperature, humidity, biological signal, and the like. The sensor 1 combines the unprocessed raw data and converts the unprocessed raw data into JSON formatted data and transmits it to the web server 2.

Thereafter, a web page URL capable of receiving and storing the information of the sensor 1 is generated in the web server 2. The web server 2 can utilize apache. The web server 2 receives the output of the sensor 1 as a PUT request based on the HTTP protocol.

Thereafter, the API driver 3 parses the JSON format data into unit data using the HDFS parsing (hasoop distributed file system parsing) library, and parses the parsed unit data so that it is optimized for the mapreduce process Are arranged in a predetermined order, and distributed and stored in the big data cluster 4 through a map de-duplication process.

Maple deuce process includes map operation and reduce operation. The map operation distributes large-scale data to several computers and processes them in parallel. A map operation is performed for each parsed unit data. The parsed unit data is read and processed in units of records, and the result is stored as key and value pairs. When storing, the same keys are sorted and stored, and the data collected by the same keys are collected and sorted by the data to be processed in the same redox operation. The Reduce operation combines the outputs of this generated map operation. The redescription operation merges and arranges the data obtained from each parsing data by key. The sorted data is stored in the big data cluster (4) by grouping them into key, value, and list pairs. Here, the key may be a value composed of one byte array of the source port number, the destination port number, the source IP address, and the destination IP address. value can be the number of packets.

2 is a schematic flowchart of a data output method between a web server and a big data cluster using an API driver according to the present invention. This is a processing flow chart when a user accesses a web page and requests big data analysis information using log data obtained from the sensor 1. [

The user logs in through the authentication API of the web server 2 such as a blog, a Twitter, or a Facebook, and accesses a web page of the corresponding URL. User authentication occurs between a consumer and a service provider, and the authentication process can be a general authentication process. The consumer requests a request token from the service provider. The service provider issues a request token to the consumer. The consumer moves the user to the service provider. Here, user authentication is performed. The service provider moves the user to the consumer. The consumer requests an access token. The service provider issues an access token. Access the user information from the consumer using the issued access token.

When the user requests the analysis data desired by the user on the web page, the API driver 3 requests the big data cluster 4 for the big data, executes the map redundancy operation from the provided big data, Suitably converts the data and outputs it to the web page. At this time, the output format may be JSON format or graph format.

As an example, when the data information collected by the sensor 1 is temperature, if the data measured in a specific section is 20, 21, and 22, the data of 20, 21, and 22 are raw data ). Digitizing the data provides the user with the desired numerical value from these raw data. For example, if the user requests an average value, the average value is converted to JSON format and displayed on the web platform. The average value is 21, the maximum value is 22, and the minimum value is 20.

In the present invention, not only the service provider can utilize the service provider to analyze the taste of the consumer, but also stores data related to the user himself / herself in the big data cluster 4 and analyzes the data stored in the big data cluster 4 in a customized manner And can be utilized as a personalized large data service system that provides information desired by the user. It is possible to provide a big data service system through an API driver 3 that can be easily accessed through a web page so that each individual can easily utilize it and can be used without changing a program for each situation .

3 is a schematic configuration diagram of a big data service system according to the present invention and includes a sensor 1, a web server 2, an API driver 3 and a big data cluster 4.

The sensor 1 senses temperature, humidity, position information, and the like, converts the raw data into a JavaScript object notation (JSON) format, and transmits the raw data to the web server 2.

The web server 2 can receive not only the output of the sensor 1 but also related data existing on the Internet to the API driver 3. [

The API driver 3 parses the data transmitted to the web server 2 in a predetermined unit using the parser library, parses the parsed unit data, arranges the parsed unit data in a predetermined order, and distributes the parsed unit data in the big data cluster 4 .

The big data cluster 4 includes a plurality of repositories, so that a replica of the data exists and parallel processing of data is performed. The big data cluster 4 performs data communication between the web server 2 and the big data cluster 4 via the API driver 3 in order to maintain high availability of a large amount of data difficult to be stored in one storage. The Big Data Cluster (4) can use a Java-based open source framework to handle big data, such as Hadoop.

The API driver 3 includes a JSON parser 5, a JSON builder 6, a user authentication module 7, a mapper 6, A deuce module 8, an analysis module 9, and a search module 10.

The JSON parser 5 parses the raw data into unit data so that the mapping deuce module 8 can operate correctly and quickly. At this time, the parsing library can be used for parsing. The JSON builder 6 combines the data with the MapReduce executed and converts it into the format requested by the user. The maple deuce module 8 performs a map operation and a reduce operation. The map operation and the redo operation are as described above.

The search module 10 selects the analysis module 9 that provides the most similar analysis method among the plurality of analysis modules 9 by classifying the user request information by theme. A plurality of analysis modules (9) selectively operate according to the search module (10), and each analysis module (9) provides different analysis methods. Therefore, by selecting the analysis module (9) corresponding to the analysis method most similar to the related topic without changing the function of the API driver (3) in various situations, the API driver (3) 3) can be provided. This can greatly improve the convenience of collecting, storing, managing and analyzing big data.

The navigation system according to another embodiment of the present invention is a traffic information providing system utilizing big data. The GPS sensor 1 is incorporated in the navigation system owned by each user. The location information or travel route of the vehicle obtained from the GPS sensor 1, The API driver 3 reads the location information of each vehicle from the web server 2, parses the unit data, and maps and de-maps the unit data to the big data cluster 4. In addition, by analyzing the big data obtained from the big data cluster 4 and providing the user with the information requested by the user, it is possible to process vast amount of data in a usable manner, and at the same time, utilizing the data obtained from each user, Information can be provided.

Another application example is a personalized health management system, which receives data such as heart rate, body temperature, body mass index, and blood glucose obtained from users' biometric sensor 1 from a corresponding web site to accumulate big data and adapt it to each user By providing data and processing, it is possible to provide a personalized health care service system utilizing scientific and systematic big data.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the invention as defined in the appended claims. You will understand.

1: Sensor 2: Web server
3: API Driver 4: Big Data Cluster
5: JSON Parser 6: JSON builder
7: User authentication module 8: MapReduce module
9: Analysis module 10: Search module
11: User authentication server

Claims (16)

The sensor converts the sensed signal into data of a JSON (javascript object notation) code and transmits it to the web server
The application programming interface (API) driver includes a step of parsing the data converted into the JSON code transmitted to the web server using the parser library, mapreduce the data, and store the parsed data in the big data cluster A method of data communication between a web server and a big data cluster using an API driver. The method according to claim 1,
After storing in the big data cluster,
Wherein the web server confirms user authentication and accepts access when a login is requested to a corresponding web page related to the output of the sensor,
The API driver analyzes the user request information input to the web page to fetch the big data accumulated in the big data cluster, analyze it according to the request of the user, convert it into data of JSON code and output it to the web page The method of claim 1, further comprising the steps of: 3. The method of claim 2,
The method of claim 1,
And converting the big data into the JSON code by mapping and de-mapping the big data. 3. The method of claim 2,
The method of claim 1,
Wherein the API driver selects an analysis module corresponding to a topic most similar to the information requested by the user among the plurality of analysis modules and operates only the analysis module to analyze the big data and output the analyzed API to the web page A method of data communication between a web server and a big data cluster. 3. The method of claim 2,
The method of claim 1,
And converting the big data into a format requested by a user on a web page and outputting the converted large data, and outputting the converted format. The method according to claim 1,
The step of mapreduce and store in the big data cluster includes:
Dividing the parsed data into key and value pairs and arranging the same key data;
And merging the data sorted by the same key and storing the data in the big data cluster by grouping them into a pair of key value and list, and transmitting the data to the big data cluster using the API driver. A sensor which converts the sensed signal into data of JSON code and transmits it to the web server;
A web server for receiving a big data analysis request signal through a user's login;
An API driver for parsing the data transmitted to the web server into unit data using a parser library, parsing the parsed unit data, arranging the parsed unit data in a predetermined order and storing the parsed unit data in a big data cluster,
And a big data cluster for distributing output data of the API driver to a plurality of repositories. 8. The method of claim 7,
The API driver,
A JSON parser for parsing the data of the JSON code into unit data using a parser library;
A JSON builder that combines the big data output from the big data cluster and
And a map de-misser for mapping the unit data output by the JSON parser to the big data cluster. 9. The method of claim 8,
The API driver,
A search module for classifying the big data analysis request signal by theme and selecting an analysis module providing the most similar analysis method among a plurality of analysis modules,
Further comprising a plurality of analysis modules that selectively operate according to the search module and provide different analysis methods. 10. The method of claim 9,
The web server comprises:
And converting the big data analysis data provided by the API driver into JSON code data according to the big data analysis request signal and outputting the JSON code data to a web page. 8. The method of claim 7,
The big data service system comprises:
Further comprising a user authentication module for allowing a user to access the web page in response to a user authentication request transmitted to a web page of the web server. A biosensor for sensing at least one of heart rate, body temperature, body mass index and blood sugar, converting the data into JSON code data, and transmitting the data to a web server;
An API driver for transmitting data of the JSON code transmitted from the bio-sensor to the web server, parsing the data, mapping the data to the big data cluster,
And a big data cluster for distributing output data of the API driver to a plurality of repositories. 13. The method of claim 12,
Further comprising a web server for receiving data on the physical condition of each user output by the biometric sensor and storing the data in the big data cluster by the API driver so that the user can utilize the big data. Personalized healthcare service system utilizing data. 14. The method of claim 13,
The web server comprises:
A big data analysis request signal of a user is inputted by a user login, and the big data analyzed by the API driver is converted into a JSON format and outputted, and the personalized health care service system using Big Data is outputted. A navigation system having a built-in GPS sensor for transmitting movement path or position information of a user vehicle to a web server;
An API driver for receiving the information transmitted from the web server and parsing and mapping the information from the web server,
And a big data cluster for distributing and storing mapleded data transmitted from the API driver. 16. The method of claim 15,
Further comprising a web server for providing the user with a web page to which the user has accessed the big data accumulated in the big data cluster by the login of the user's web page.

Images