KR20090000932A - Processing system and method for large capacity data from the remote sensor - Google Patents

Abstract

A massive data processing system and a method measured in a remote site sensor installed in ubiquitous-city space are provided to integrate a sensor and a controller which is locally managed and to process the massive data safely. A gateway(300) removes data which is overlapped with past data of data received by each protocol from a plurality of sensors. Data of different network protocol which is not overlapped with the past data is recomposed as a standardized protocol. A load balancer part(200) distributes data received from the gateway. A platform(100) outputs so that the operator clearly recognizes event data which the load balancer part processes. The gateway determines that current data value is overlapped if the current data value is included within specific error range of the past data value. The load balancer part includes a plurality of data processing servers(230-1, .,230-n) by analyzing the data.

Description

Mass data processing system measured by remote sensor and its method {PROCESSING SYSTEM AND METHOD FOR LARGE CAPACITY DATA FROM THE REMOTE SENSOR}

1 is a network configuration diagram schematically showing a large data processing system measured by a U-City remote sensor according to an embodiment of the present invention.

2 is a structural diagram showing a large data processing system measured by a U-City remote sensor according to an embodiment of the present invention.

3 is a flowchart illustrating a method of processing a large amount of data measured by a U-City remote sensor according to an exemplary embodiment of the present invention.

The present invention relates to a large-capacity data processing system measured by a remote sensor and a method thereof, and more particularly, to a system and a method for processing the large-capacity data measured by a sensor at a remote location installed in a ubiquitous city space. It is about.

Ubiquitous city (hereinafter referred to as "U-City") innovates existing information infrastructure and integrates ubiquitous technology into infrastructure to make urban functions and management more efficient, cope with real-time situations in cities, and provide information and communication services. It is a new concept city that provides a convenient and safe life. For this U-City, various kinds of sensors are installed in many urban infrastructures and living spaces, and they are configured and utilized in a structure suitable for service through a combination with a controller that converts them into electrical signals and communicates with them. .

In the related art, a separate controller is configured for each sensor, and services are provided in a distributed manner through a separate protocol according to a manufacturer or a use of the sensor, thereby becoming a problem of management efficiency. Therefore, there is a need for a policy that integrates and manages distributed equipment, and accordingly, a system capable of stably processing a large amount of integrated data is required.

Accordingly, an object of the present invention is to provide a system having a structure capable of integrating and managing heterogeneous sensors and controllers that are locally distributed and managed to collect and safely process large data.

In order to solve the above technical problem, a large-capacity data processing system measured by a remote sensor according to an embodiment of the present invention,

A gateway for filtering and removing data overlapping with the past data among the data received by the heterogeneous protocols from the plurality of sensors, and reconstructing the data of the heterogeneous protocol that does not overlap with the past data into a standardized protocol; A load balancer for distributing data received from the gateway; And a platform for outputting the event data processed by the load balancer so that an operator can recognize the event data.

Here, the gateway determines that the current data value is overlapped if it falls within a specific error range of the past data value.

On the other hand, the method for processing large data measured in a plurality of heterogeneous sensors according to an embodiment of the present invention,

a) receiving data of heterogeneous protocols received from the plurality of heterogeneous sensors, and filtering and removing data duplicated with past data among the received data; b) reconstructing heterogeneous protocol data that is not duplicated with the historical data into a standardized protocol on a system; c) allocating the data reconstructed in step b) to a plurality of data processing servers; And d) analyzing the event for the allocated data and storing the event in a database.

And e) displaying the event data analyzed in step d) together with the normal data and the data associated with the event data.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present invention. In the drawings, parts irrelevant to the description are omitted in order to clearly describe the present invention, and like reference numerals designate like parts throughout the specification.

Throughout the specification, when a part is said to "include" a certain component, it means that it can further include other components, without excluding other components unless specifically stated otherwise. In addition, the terms “… unit”, “… unit”, “module”, “block”, etc. described in the specification mean a unit that processes at least one function or operation, which is hardware or software or a combination of hardware and software. It can be implemented as.

Now, a mass data processing system and a method thereof measured by a remote sensor according to an exemplary embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 is a network configuration diagram schematically showing a large data processing system measured by a U-City remote sensor according to an embodiment of the present invention.

Referring to FIG. 1, in the embodiment of the present invention, the controller 20 receives a signal periodically transmitted from the remote sensor 10 installed in the urban space of U-City, and the appropriate sensor data from the controller 20. Converted to and delivered to the gateway 300. The gateway 300 periodically removes data overlapping with the past data among the sensor data received through filtering, and reconfigures the non-overlapping sensor data into a standardized protocol and transmits the data to the load balancer 200. In addition, the load balancer unit 200 provides a large amount of data to be distributed and read out to the platform 100 by appropriately distributing and reading a plurality of data processing servers, so that an operator can manage the U-City through the platform 100. Identify and respond to them.

Meanwhile, the structure of a large data processing system according to an embodiment of the present invention will be described with reference to FIG. 2.

2 is a structural diagram showing a large data processing system measured by a U-City remote sensor according to an embodiment of the present invention.

Referring to FIG. 2, a large-scale data processing system measured by a remote sensor according to an exemplary embodiment of the present invention includes a heterogeneous sensor (not shown) 10, a controller (not shown 20), a gateway 300, and a load balancer unit. 200 and platform 100.

The sensor 10 is a sensor installed in the urban space of the U-City and has a function of sensing physical phenomena occurring in the urban space and transmitting the same to the controller 20.

The physical phenomenon may be, for example, temperature, humidity, wave height, wind direction, speed, and the like, and each performs various measurement functions according to the type of the sensor 10. In addition, the sensor 10 has a unique protocol different from each other and has a function of delivering the collected sensor data to the controller 20 in cycles.

The controller 20 is a concentrator that collects sensor data collected from heterogeneous sensors 10, and includes DI (Digital Input), AI (Analog Input), DO (Digital Output), Ethernet, and SI (Serial Input) ports. It consists of The controller 20 may be connected to about 1 to 41 sensors 10, and transmits sensor data collected from the connected sensors 10 (hereinafter referred to as “data” for convenience) to the gateway 300. Has the function to Descriptions of the ports are omitted in the general understanding level of those skilled in the art.

The gateway 300 serves as a gateway for delivering data received from the plurality of controllers 20 connected to the remote sensors 10 to the load balancer 200, and includes a filter 310 and a data reconstruction unit ( 320).

The filter unit 310 receives data of the controller 20 managed for each communication channel from the gateway 300 to store state information (ON / OFF) of the sensor 10, and compares the received sensor data with past data. If it overlaps with the past data, it has the function of filtering. In more detail, the filter unit 310 calculates an error rate through an arithmetic operation of data repeatedly received by the gateway 300, and loads redundant redundant data by filtering the overlapping data within the error range. Transmission to the balancer 200 may be omitted.

For example, if the data received from any analog sensor 10 generates 0.2% error, the measured value of 100.5 is repeatedly received in the range of 100.3 to 100.7, etc. do. When the data after the transmitted data changes within the error range, the network occupancy rate may be lowered by not transmitting the change history.

That is, the filter unit 310 has a feature of efficiently processing a large amount of data by minimizing a phenomenon that the network occupancy increases by periodically transmitting data without determining whether the data is changed in a conventional large data system.

The data reconstruction unit 320 has a function of analyzing data received through heterogeneous protocols of heterogeneous sensors 10 located at a remote location, and reconstructing data received by a unique protocol into one standardized protocol. Therefore, it is possible to integrate and manage the station devices distributed in the remote location in the platform (100).

The load balancer unit 200 includes an active server 210, a passive server 220, a data processor 230 and a database 240 including a plurality of data data processing servers 230-1,..., 230-n. ), It has a function to distribute a large amount of data received.

The active server 210 and the passive server 220 collect data from a plurality of gateways 300 installed in the field and store the data in the database 240, and the plurality of data processing servers 230-1,..., 230-n Is assigned to). At this time, in the normal operation state of the system, the active server 210 and the passive server 220 do not work at the same time, only the active server 210 serves, and the passive server 220 is the active server 210. ) Monitors the proper function and operates when the active server 210 exceeds or fails.

The active server 210 includes a data processing server manager 211 and an event provider 212.

The data processing server manager 211 determines the share of each data processing server 230-1,..., 230-n of the data processing unit 230, and has a low share of the data received from the gateway 300. Has the function of assigning in order.

The event provider 212 provides the platform 100 so that the system operator can recognize the event providing unit 212 if it is determined that the data generates the event. When the event provider 212 confirms that an abnormally measured value is received from the remote sensors 10, the power supply of the sensor 10 is incomplete, or the physical abnormality is in a range that the sensors 10 manage. If it is, it is judged to be provided by the system operator.

The passive server 220 is a server for monitoring and backing up the active server 210. The passive server 220 monitors whether the active server 210 has failed and if the failure of the active server 210 is expected, the function can be transferred to process the job. It is configured, and since the configuration and operation overlap with the active server 210, a detailed description thereof will be omitted.

The data processor 230 includes first data processing servers 230-1 to n-th data processing servers 230-n, and the first data processing server 230-1 includes a data load unit 231 and an event. It includes a processing unit 232, a primary data queue 233, a secondary data queue 234, and an occupancy check unit 235, and other data processing servers 230-n have the same configuration.

The data load unit 231 imports data allocated from the active server 210 from the database 240.

The primary data queue 233 is a kind of buffer for temporarily storing data stored in the database 240 by the data load unit 231, and the secondary data queue 234 stores an event of data in the event processor 232. A certain amount of data is periodically taken from the primary data queue 233 and transmitted to the event processor 232 for analysis. In this case, the size of the buffer is checked according to the set reference value of the secondary data queue 234, and when the size is 1 or more, data is obtained from the primary data queue 233.

For example, when the door is closed by the door sensor entering and exiting the RFID card, the normal data may be detected by the sensor 10, and the normal data may be filtered by the redundant data on the gateway 300. At this time, when someone accesses the door using the authorization card, if the door is opened and closed, a single event occurs, where the single event also has a value in which the size of the buffer does not exceed 1 as usual data.

On the other hand, if the event that the door is opened or the door is broken and opened when the authorized card is not used, this is the event processing unit 232 by the secondary data queue 234 with data whose size of the buffer exceeds 1; Is sent to. In this case, data having a buffer size of 1 or more may be analyzed in association with other event data received at an event occurrence time. For example, the event situation may be output from the platform 100 through additional data such as a camera photographing the corresponding doorway or a moving sensor installed at the corresponding entrance.

Such data queues limit data capacity because the data queue cannot allocate an unlimited amount of data in the memory management dimension of the system. Accordingly, the primary data queue 233 and the event managed by the data load unit 231 in order to minimize the occurrence of an exception due to the processing time difference between the data load unit 231 and the event processor 232 become full. The secondary data queue 234 managed by the processor 232 is configured to process a large amount of data.

The event processing unit 232 analyzes the event level for each occurrence or risk of the data obtained from the secondary data queue 234 to process the data so that the user can easily recognize the data, and the data generating the event to the database 240. It serves to store.

The occupancy checker 235 checks the size of the primary data queue 233 and the size of the secondary data queue 234 of the first data processing server 230-1 and transmits the occupancy state to the active server 210.

The platform 100 functions to control and manage a central system that manages the sensors 10 at a remote place of the U-City, and outputs the data received from each sensor 10 so that the system operator can check the data. . In addition, when the platform 100 requires event data linked to one event occurrence, it is possible to reduce the process of searching the database 240 one by one and output the threaded information through the data queue. It is effective to manage the system. In addition, by immediately responding to the event information reported in real time from a remote location it can perform the role of a dynamic U-City management system.

Meanwhile, a method of processing large data measured by a remote sensor of U-City according to an embodiment of the present invention will be described with reference to FIG. 3.

3 is a flowchart illustrating a method of processing a large amount of data measured by a U-City remote sensor according to an exemplary embodiment of the present invention.

Sensors 10 installed in the A region of U-City sense the physical phenomenon in real time and transmit data to the controller 20. Here, the physical phenomenon may be a temperature, a gas, a wind direction, a speed, and the like. In the embodiment of the present invention, it is assumed that a fire occurs in the A region.

Referring to FIG. 3, the gateway 300 receives data transmitted from the sensors 10 in the region A from the controller 20 (S301). The gateway 300 determines whether the received data is normal data (S302), reconfigures the protocol standardized in the system (S303), and transmits the data to the active server 210 (S304).

The active server 210 determines the occupancy of the data processor 230 (S305), and as a result, allocates the data to the first data processing server 230-1, which is determined to have a low occupancy, and stores the data in the database 240. (S306, S307).

The first data processing server 230-1 loads the allocated data from the database 240 (S308) and stores the allocated data in the primary data queue 233 (S309). The secondary data queue 234 takes data from the primary data queue (S310) and analyzes events for each sensor through the event processor 232 (S311). When the fire occurs, the values of the sensors for detecting the temperature and the gas are out of the normal level. Thus, the event analysis result of the sensor 10 is stored in the database 240 (S312) and transmitted to the active server. (S313). At this time, preferably, the system operator displays the risk level based on the result of analyzing the event data of the sensor 10.

When the active server 210 transmits the event occurrence situation to the platform (S314), the platform 100 outputs the received event situation so that the system user can recognize it (S315). In this case, the platform 100 may be output through an alarm using an image or a sound, and may be provided to compare and monitor data and event data that are normally measured, or a fire range and progress as linked event information to identify a site situation. Additional real-time situation identification and countermeasures such as direction, speed, and on-site confirmation through video can be performed.

Through the large-capacity data processing system according to an embodiment of the present invention, it is possible to overcome the problems of conventional heterogeneous data collection and local data management limits, and to collect and process large-capacity data stably in a wide range.

Although the embodiments of the present invention have been described above, the present invention is not limited only to the above embodiments, and various other changes are possible.

For example, in the embodiment of the present invention illustrated in FIG. 2, one active server 210 performs a load balancing function and configures a passive server 220, but may include two or more passive servers. The server 220 may be driven simultaneously with the active server 210.

In addition, the filter unit 310 of the gateway 300 illustrated in FIG. 2 may omit data filtering under the operator's control. For example, the specific sensor 10 or the controller 20 may check or periodically check whether the normal data is transmitted in the ON state.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements of those skilled in the art using the basic concepts of the present invention defined in the following claims are also provided. It belongs to the scope of rights.

By the above-described configuration, there is an effect of overcoming the limitations of data collection and heterogeneous data management limits through heterogeneous sensors, and collecting and managing heterogeneous data globally.

In addition, there is an effect that can efficiently process data by providing a stable system through the gateway filtering and the load balancer for the large-capacity data collected globally.

Claims (13)

In a system for processing a large amount of data measured by a number of heterogeneous sensors, A gateway for filtering and removing data overlapping with the past data among the data received by the heterogeneous protocols from the plurality of sensors, and reconstructing data of the heterogeneous protocol that does not overlap with the past data into a standardized protocol; A load balancer for distributing data received from the gateway; And A platform for outputting the event data processed by the load balancer so that the operator can recognize Large data processing system comprising a. The method of claim 1, The gateway, If the current data value is within a specific error range of the past data value, it is determined that the duplicate data processing system. The method of claim 1, The load balancer unit, A plurality of data processing servers analyzing the data to analyze whether an event has occurred; One or more load balancing servers for allocating data received from the gateway to the plurality of data processing servers; And Database that stores the data Large data processing system comprising a. The method of claim 3, wherein The load balancing server, A data processing server manager configured to identify a share of the plurality of data processing servers and allocate data to a data server having a low share; And Event provider to provide the platform when the event data is confirmed from the data processing server Large data processing system comprising a. The method according to claim 1 or 4, The load balancing server, And one or more passive servers configured to perform load balancing by monitoring an operation of the active server and driving when a failure occurs. The method of claim 5, wherein The data processing server, A data load unit which loads allocated data from the load balancing server; A primary data queue which is a buffer for temporarily storing the loaded data; An event processor that analyzes an event level for each occurrence and risk of the data; A secondary data queue, which periodically takes a certain amount of data from the primary data queue and delivers it to the event processor; And Occupancy check unit for monitoring the state of the primary data queue and the secondary data queue, and transmits to the active server Large data processing system comprising a. The method of claim 1, The platform, And outputting the data generating the event by comparing the data with the normal data and searching for data associated with the data generating the event. In the method for processing large data measured by a plurality of heterogeneous sensors, a) receiving data of heterogeneous protocols received from the plurality of heterogeneous sensors, and filtering and removing data overlapping with past data among the received data; b) reconstructing heterogeneous protocol data that is not duplicated with the historical data into a standardized protocol on a system; c) allocating the data reconstructed in step b) to a plurality of data processing servers; And d) analyzing the event for the allocated data and storing it in a database Large data processing method comprising a. The method of claim 8, e) displaying the event data analyzed in step d) together with the normal data and the data associated with the event data. The method of claim 8, Step a) is And determining that the current data value is overlapped when included in a specific error range of the past data value. The method of claim 8, Before step c), c-1) checking a share of each data processing server; And c-2) identifying a data processing server having the lowest share among the data processing servers; Large data processing method comprising a. The method according to any one of claims 8 to 11, Before step d), d-1) loading the allocated data; d-2) temporarily storing the loaded data in a primary data queue; And d-3) allowing the secondary data queue to retrieve and analyze a predetermined amount of the temporarily stored data. Large data processing method comprising a. The method of claim 12, In step d), A method of processing a large amount of data, characterized in that the level of danger is indicated in the data that caused the event.

Images