TWI626548B - Data collection and storage system and method thereof - Google Patents

Abstract

A data collection and storage system for collecting and storing plural message data from at least one user device, the data collection and storage system mainly comprising a data collection subsystem, the data collection subsystem comprising at least one data mediation module At least one event processing module and at least one event loading module, the data mediation module includes a first data area and a second data area, the information data is stored in the first data area, the event The processing module reads the message data from the first data area of the data broker module, and then obtains at least one data segment based on the message data and a predetermined condition, and writes the data segment to the data broker module In the second data area, the event loading module periodically reads the data segment from the second data area of the data broker module.

Description

Data collection and storage system and method thereof

The present invention relates to a data collection and storage system and method thereof, and more particularly to a data collection and storage system and method thereof for use in Big Data.

In recent years, with the increasing use of big data, technologies and computing platforms related to big data processing, such as Hadoop, have also flourished. Hadoop is the Open Source Project under the Apache Software Foundation, which provides a decentralized computing environment and storage space for large amounts of data.

Please refer to Figure 1. A well-known accurate recommendation platform 1 is used. For example, the personalized precision recommendation platform 1 is an accurate recommendation and consumer behavior analysis platform (Etu Recommender), which is built on the Hadoop platform. The precision recommendation platform 1 is configured to receive message data from the plurality of user devices 2, and provide at least one recommendation list based on the message materials. The personalized precision recommendation platform 1 includes a load balancer (11), a data collection system 12, a Hadoop Distributed File System (HDFS) 13, a data conversion processing unit 14, and a The recommendation algorithm processing unit 15; wherein the data collection system 12 includes a plurality of servo devices 121, each of the server devices 121 includes a web server 122 and a local file system 123. The web server 122 is implemented using NGINX.

The message data from the user devices 2 is transmitted to the server devices 121 of the data collection system 12 via the load balancer 11 by a GET method, and the message data includes a plurality of data keys/ Key/Value Pairs; the website server 122 of each server 121 receives the message data and writes it to the local file system 123; the data collection system 12 periodically (eg, each Hour) The collected message data is written into the Hadoop distributed file system 13; the data conversion processing unit 14 performs Hadoop capture-convert-load on the content of the Hadoop distributed file system 13 (Extract-Transform-Load) , referred to as ETL), to convert the data into a format that can be imported into the database; the recommendation algorithm processing unit 15 performs computational analysis based on the content of the Hadoop distributed file system 13 (eg, data correlation analysis, data similarity) Degree analysis) to give the recommendation list, wherein the recommendation list includes product recommendation information, content recommendation information, on-site advertisement recommendation information, and the like.

The data collection system 12 of the personalized accurate recommendation platform 1 only collects the message data, and then periodically writes the collected message data to the Hadoop distributed file system 13 without performing any information on the message data. Conversion or processing makes the subsequent application of the message material more limited and not convenient.

Accordingly, it is an object of the present invention to provide a data collection and storage system suitable for collecting and storing plural message data from at least one user device, the data collection and storage system primarily comprising a data collection subsystem, The data collection subsystem includes at least one data broker module, at least one event processing module, and at least one event loading module, wherein the data broker module includes a first data area, a second data area, and a third Data area, such information The data is stored in the first data area, wherein each message data corresponds to an event occurrence time; the event processing module is electrically connected to the data mediation module for the first data from the data mediation module The area reads the message data, and then obtains at least one data segment according to the message data and a predetermined condition, and writes the data segment to the second data area of the data mediation module, wherein the event processing module Including at least one pre-processing unit, the pre-processing unit is configured to perform data pre-processing according to the message data to obtain a corresponding plurality of available data, and the event processing module further writes the available data into the data mediation module. The third data area is electrically connected to the data mediation module for periodically reading the data segment from the second data area of the data mediation module.

Another object of the present invention is to provide a data collection and storage method suitable for collecting and storing plural message data from at least one user device, the data collection and storage method being executed by a data collection and storage system. The data collection and storage system includes a data collection subsystem and a distributed file subsystem. The data collection subsystem includes a data mediation module, and the data mediation module includes a first data area and a second data area. And the third data area, the data collection and storage method comprises the steps of: (a) receiving the message data and storing the message data in the first data area of the data mediation module; (b) from the The first data area of the data broker module reads the message data; (c) obtaining at least one data segment according to the message data and a predetermined condition, wherein the step (c) comprises the following sub-steps: (c-1) Pre-processing the data according to the information to obtain the corresponding plural available data, and (c-2) sorting the available data corresponding to the information according to the time of occurrence of the events, and (i) writing the data segment to the second data area of the data broker module; and (e) periodically reading from the second data area of the data broker module Take the data segment and write the data segment to the score Bulk file subsystem.

The effect of the present invention is that the event processing module obtains the data segment according to the message data and the preset condition, and then the event loading module periodically writes the data segment to the distributed file subsystem, increasing The scope and convenience of such information materials in the subsequent application.

3‧‧‧ Data collection and storage system

4‧‧‧User device

5‧‧‧Data collection subsystem

51‧‧‧ format check module

52‧‧‧Data mediation module

521‧‧‧First data area

522‧‧‧Second data area

523‧‧‧ Third data area

524‧‧‧data segment

53‧‧‧ Event Processing Module

531‧‧‧Pretreatment unit

54‧‧‧ Event Loading Module

55‧‧‧Control unit

6‧‧‧Distributed file subsystem

7‧‧‧Data Analysis Application Subsystem

71‧‧‧Application unit

S1~9‧‧‧ steps

Other features and effects of the present invention will be apparent from the following description of the drawings, wherein: FIG. 1 is a block diagram illustrating one of the conventional precision recommendation platforms; FIG. 2 is a block diagram illustrating the present invention. A first preferred embodiment of a data collection and storage system; FIG. 3 is a flow chart illustrating a data collection and storage method corresponding to the first preferred embodiment; and FIG. 4 is a block diagram illustrating the data collection of the present invention And a second preferred embodiment of the storage system; and FIG. 5 is a block diagram illustrating a third preferred embodiment of the data collection and storage system of the present invention.

The above and other technical contents, features and advantages of the present invention will be apparent from the following detailed description of the preferred embodiments of the invention.

Before the present invention is described in detail, it should be noted that in the following description, similar or equivalent elements are denoted by the same reference numerals.

Referring to FIG. 2, a first preferred embodiment of the data collection and storage system 3 of the present invention The embodiment is applicable to collecting and storing the plurality of message data from the at least one user device 4, the data collection and storage system 3 is electrically connected to the user device 4; the data collection and storage system 3 includes a data collector The system 5 is electrically connected to the distributed file subsystem 6 of the data collection subsystem 5, and is electrically connected to the data collection subsystem 5 and the data analysis application subsystem 7 of the distributed archive subsystem 6.

In the first preferred embodiment, the distributed file subsystem 6 is a Hadoop distributed file subsystem; the data analysis application subsystem 7 includes at least one application unit 71, wherein the data analysis application subsystem 7 can For a server having various specific applications or functions, the application unit 71 can be implemented in a software manner, and the implementation aspect is an application built in the data analysis application subsystem 7, and the application unit 71 is also It can be implemented in a firmware or hardware manner, and is not limited to the first preferred embodiment.

The data collection subsystem 5 includes a format check module 51, a data mediation module 52 electrically connected to the format check module 51, and an event processing module electrically connected to the data mediation module 52. The group 53 is electrically connected to an event loading module 54 of the data broker module 52. The data mediation module 52 includes a first data area 521, a second data area 522, and a third data area 523.

The format check module 51 of the data collection subsystem 5 receives the message data from the user device 4, and performs format check on the message data. The formatted message information is transmitted to the data collection. The data mediation module 52 of the subsystem 5 is stored in the first data area 521. In the first preferred embodiment, the message data is transmitted from the user device 4 to the data processing system 5 by the POST method. Further, the message data is placed in the JSON format. Transmitted to the data collection within the message-body Subsystem 5.

The event processing module 53 reads the message data from the first data area 521 of the data mediation module 52. Each message data corresponds to an event occurrence time, and the event processing module 53 is based on the message data and At least one data segment 524 is obtained by a predetermined condition, and the data segment 524 is written to the second data region 522 of the data broker module 52.

In the first preferred embodiment, the event processing module 53 includes at least one pre-processing unit 531, and the pre-processing unit 531 of the event processing module 53 is configured to perform data according to the message data. Pre-processing to obtain the corresponding plural available data. For example, the pre-processing unit 531 can obtain the available data of the country, the source, and the like of the IP from the IP address (IP Adress) in the message data; wherein the pre-processing unit 531 can view the data analysis application of the back end. The requirements of subsystem 7 are developed or extended through physical interface. Further, in the first preferred embodiment, the preset condition is related to the event occurrence time. In other words, the event processing module 53 generates the available data corresponding to the message data according to the event. The time is sorted, and then merged into the data segment 524 in units of a time interval (for example, hourly), and then written into the second data area 522 of the data mediation module 52; in addition, the event processing module 53 further The available data that is not sorted according to the event occurrence time can be directly written into the third data area 523 of the data mediation module 52; and the event loading module 54 periodically (for example, every hour) from the data. The second data area 522 of the intermediation module 52 reads the data segment 524 and writes the data segment 524 to the distributed archive subsystem 6.

The data analysis application subsystem 7 is configured to perform post-processing of the data according to the data segment 524 of the distributed file subsystem 6 or the available data in the third data area 523 of the data broker module 52; Post-processing includes data statistics performed by the application unit 71, Analysis, induction, etc. In the first preferred embodiment, when the application unit 71 of the data analysis application subsystem 7 needs to perform post-processing of the data according to the available data of the instant update, the data analysis application subsystem 7 can directly Reading from the third data area 523 of the data broker module 52; when in the application unit 71 of the data analysis application subsystem 7, there is a data segment 524 that needs to be aggregated according to each time interval. When performing statistics and analysis, the data analysis application subsystem 7 can read the data segment 524 required by the distributed archive subsystem 6. Further, in the first preferred embodiment, the distributed file subsystem 6 is a Hadoop distributed file subsystem. Since the Hadoop distributed file subsystem does not support the random access method, only The batch access mode is supported; in addition, when each message data is generated (corresponding to the event occurrence time), a temporary disconnection may occur. If the available data is not aggregated into the data segment 524, the data cannot be guaranteed. The available data in the mediation module 52 is stored according to the corresponding event occurrence time. When the application unit 71 of the data analysis application subsystem 7 needs the available data in a certain time interval, it is impossible to know that it needs to be The distributed file subsystem 6 continuously reads the data of the amount of offset (Offset); therefore, merging the available data into the data segment 524 allows the application unit 71 of the data analysis application subsystem 7 to read the data on the data. For convenience.

In addition, in the first preferred embodiment, the data transmission between the components on the network is compressed to reduce the overall network traffic load.

Referring to FIG. 2 and FIG. 3, the data collection method corresponding to the first preferred embodiment includes the following steps: receiving the message data from the user device 4 and performing the message as shown in step S1. Data is formatted; as shown in step S2, the correct format of the message data is stored in the data. The first data area 521 of the mediation module 52; as shown in step S3, the message data is read from the first data area 521 of the data mediation module 52; as shown in step S4, according to step S3 And reading the available information to obtain the corresponding available data; as shown in step S5, writing the available data to the third data area 523 of the data broker module 52; as shown in step S6, according to the The preset condition merges the available data into the data segment 524; as shown in step S7, the data segment 524 is written into the second data region 522 of the data broker module 52; and as shown in step S8, timing The data segment 524 is read from the second data area 522 of the data broker module 52, and the data segment 524 is written to the distributed archive subsystem 6.

As shown in step S9, the data is post-processed according to the data segment 524 of the distributed file subsystem 6, or the available data in the third data area 523 of the data broker module 52.

Referring to FIG. 4, a second preferred embodiment of the data collection and storage system 3 of the present invention is electrically connected to a load balancer 8; the load balancer 8 is electrically connected to at least one user device 4. The data collection and storage system 3 includes a data collection subsystem 5, a distributed file subsystem 6, and a data analysis application subsystem 7; wherein the data collection subsystem 5 includes a complex format check module 51, electrical a plurality of data mediation modules 52 connected to the format check modules 51, a plurality of event processing modules 53 electrically connected to the data mediation modules 52, and electrical connections The plurality of event loading modules 54 of the data broker modules 52.

The operations of the format check module 51, the data mediation modules 52, the event processing modules 53, and the event loading modules 54 are similar to the first preferred embodiment, so The implementation details of the components are described in detail, and only the differences in their configurations are described.

The load balancer 8 is configured to receive the plurality of message data from the user device 4 and distribute the message data to the format check module 51 for format check. Preferably, each data mediation module 52 It is integrated with an event loading module 54 in the same device to further reduce the overall network traffic load. In the second preferred embodiment, the data mediation module 52 and the event loading module 54 are implemented in a Kafka Queues manner. Each data mediation module 52 is a Kafka server. (Kafka-Broker).

Referring to FIG. 5, a third preferred embodiment of the data collection and storage system 3 of the present invention is electrically connected to at least one user device 4, and the data collection and storage system 3 receives a plurality of components from the user device 4. The data collection and storage system 3 includes a data collection subsystem 5, a distributed file subsystem 6, and a data analysis application subsystem 7; wherein the data collection subsystem 5 includes a format check module 51. A data mediation module 52 electrically connected to the format check module 51, a control unit 55 electrically connected to the data mediation module 52, electrically connected to the control unit 55 and the data mediation module 52 An event processing module 53 is electrically connected to an event loading module 54 of the data broker module 52. The data mediation module 52 includes a first data area 521, a second data area 522, and a third data area 523. The similarities with the first and second preferred embodiments described above are not described again, and only the differences will be described below.

In the third preferred embodiment, the control unit 55 is configured to perform condition setting and provide the set conditions to the event processing module 53 for related processing. For example In the control unit 55, a filter condition may be set for the event processing module 53 to filter the message data from the first data area 521 of the data broker module 52. The control unit 55 may also be configured. a preset condition for the event processing module 53 to obtain at least one data segment 524 according to the message data and the preset condition, and write the data segment 524 to the second data region of the data mediation module 52 522; In a broad sense, the event processing module 53 performs related processing according to the conditions set by the control unit 55, and can obtain different data segments 524 of the group, so that the application range of the message materials is wider and more convenient.

In summary, please refer to FIG. 2, FIG. 4 and FIG. 5. The data collection and storage system 3 of the present invention optimizes data collection and storage of big data, and its functions are as follows: First, the information is based on the POST method. The content of the message data can be richer and more diverse, and the data transmitted by the POST method is also more secure; secondly, the event processing module 53 is based on the transmission. The message data is preprocessed to obtain the available data, and the available data is aggregated into the data segment 524 according to the preset condition, and the event loading module 54 periodically writes the information. Decentralized file subsystem 6, and the data analysis application subsystem 7 can read the available data directly from the third data area 523 of the data broker module 52, or read the dispersion, depending on the requirements of its application unit 71. The data segment 524 in the file subsystem 6 greatly increases the application range and convenience of the message data; third, the data transmission between the components on the network is compressed, which can reduce the overall network. Road flow A load; it can really achieve the object of the present invention.

However, the above is only the embodiment of the present invention, and the scope of the invention is not limited thereto, and all the equivalent equivalent changes and modifications according to the scope of the patent application and the patent specification of the present invention are still The scope of the invention is covered.

Claims (15)

A data collection and storage system for collecting and storing plural message data from at least one user device, the data collection and storage system comprising: a data collection subsystem, comprising: at least one data mediation module, including a first a data area, a second data area, and a third data area, wherein the message data is stored in the first data area, wherein each message data corresponds to an event occurrence time; at least one event processing module, electricity The event processing module reads the message data from the first data area of the data mediation module, and obtains at least one data segment according to the message data and a predetermined condition. And writing the data segment to the second data area of the data mediation module, where the event processing module includes at least one preprocessing unit, and the preprocessing unit is configured to perform data preprocessing according to the message data to obtain a corresponding The plurality of available data, the event processing module also writing the available data to the third data area of the data broker module; and at least one event loading module Electrically connected to the intermediary data module, the module load event from time to time to the second data area of the information intermediary module reads the data segment. The data collection and storage system of claim 1 , wherein the data collection subsystem further comprises at least one format check module electrically connected to the data broker module, wherein the format check module is configured to use the information module Performing a format check and storing the correctly formatted message data in the first data area of the data mediation module. The data collection and storage system of claim 1, wherein the data collection subsystem further comprises a control unit electrically connected to the data broker module and the event processing module for setting the preset condition. The data collection and storage system of claim 1, further comprising a distributed file subsystem, wherein the distributed file subsystem is electrically connected to the event loading module of the data collection subsystem for receiving and storing This event loads the data segment transmitted by the module. The data collection and storage system of claim 4, further comprising a data analysis application subsystem electrically connected to the data collection subsystem and the distributed file subsystem, wherein the data analysis application subsystem is configured to be based on the dispersion The data segment of the file subsystem or the available data in the third data area of the data mediation module is processed and processed. The data collection and storage system of claim 5, wherein the data analysis application subsystem comprises at least one application unit. The data collection and storage system of claim 1, wherein the data mediation module and the event loading module are integrated in the same device. The data collection and storage system of claim 1, wherein each data brokering module is a Kafka server. A data collection and storage method for collecting and storing plural message data from at least one user device, the data collection and storage method being performed by a data collection and storage system, the data collection and storage system comprising a dispersion And a data collection subsystem, the data collection subsystem comprising a data mediation module, the data mediation module comprising a first data area, a second data area, and a third data area, the data Collection and storage method package The method includes the following steps: (a) receiving the message data and storing the message data in the first data area of the data broker module; (b) reading the first data area of the data broker module (c) obtaining at least one data segment based on the message data and a predetermined condition, wherein the step (c) comprises the following sub-steps: (c-1) performing data pre-processing according to the message data to obtain a correspondence And (c-2) sorting the available data corresponding to the information according to the time of occurrence of the events, and merging the data segments into units of time intervals; (d) the data segments Writing to the second data area of the data broker module; and (e) periodically reading the data segment from the second data area of the data broker module and writing the data segment to the distributed file subsystem. The method for collecting and storing data according to claim 9 further includes the following steps before step (a): (f) performing a format check on the message data; and (g) storing the message data in the correct format. The first data area of the data mediation module. The method for collecting and storing data according to claim 9, wherein the preset condition described in the step (c) is related to the occurrence time of the plurality of events corresponding to the message material. The method for collecting and storing data according to claim 9 further includes the following steps after sub-step (c-1): (h) writing the available data to the third data area of the data broker module. The method for collecting and storing data according to claim 12, further comprising the following steps after step (e): (i) according to the data segment of the distributed file subsystem, or the third data of the data broker module The available data in the area is used for post processing of the data. The method of collecting and storing data according to claim 9, wherein the message data of step (a) is transmitted by the user device to the data processing subsystem by a POST method. The method of collecting and storing data according to claim 9, wherein the message data of step (a) is transmitted to the data collection subsystem in a message-body by using a JSON format.