KR101511338B1 - Device for aggregating data, device for managing data and system for aggregating data - Google Patents

Abstract

A data collection apparatus according to the present invention includes a data receiving unit which receives data from a collection object, a data transmitting unit which transmits the received data to a passing device, and a control unit which controls the operation of the data transmitting unit according to a preset transmission path. The transmission path includes a basic transmission path and a load transmission path. The control unit controls the data transmitting unit to operate along the basic transmission path.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a data collection device, a data collection management device, and a data management system,

The present invention relates to a data collection device, a data collection management device, and a data management system.

Unstructured data, which is difficult to manage compared to existing data, is generally referred to as big data because the generated amount, generation cycle, and generation format are vast and varied. Such big data can be variously generated in a digital environment as computer processing technology develops. Big Data is considered to be able to improve productivity in the public, manufacturing, retail, and medical sectors, and is regarded as a key resource that determines future competitiveness, and research on accurate understanding and effective utilization is actively being carried out.

Big data is collected and stored for batch analysis. However, the analysis system for big data can not operate unless the desired data is collected and transmitted appropriately in the desired manner. In addition, when the number of physical systems to be collected increases, and real-time processing of various logs occurring in various systems is required, it is important to perform data collection accurately and efficiently and stably manage the collection process.
[Prior Art Literature]
Korean Published Patent Application No. 10-2013-0140508 (Publication date: December 24, 2013)

An object of the present invention is to provide a management solution that can accurately and efficiently collect big data from each collection object and deliver it to a storage device as a target device and to provide a stable management solution for the collection process.

A data collecting apparatus according to an exemplary embodiment of the present invention includes a data receiving unit for receiving data from a collection target, a data transmitting unit for transmitting the received data to the passing equipment, and a control unit And a control unit.

The transmission path includes a basic transmission path, and the control unit can control the data transmission unit to operate according to a basic transmission path.

The transmission path may include a failure transmission path, and when a network failure occurs between the data acquisition device and the transit device, the control unit may control the data transmission unit to operate according to the failure transmission path.

The transmission path may include a load transmission path, and if the load amount is larger than the load amount reference set in the data collection device, the control unit may control the data transmission unit to operate according to the load transmission path.

The transmission path may include a data value transmission path that is set differently according to the characteristic value of the data, and the control unit may control the data transmission unit to operate according to the data value transmission path.

And a storage unit for storing the data received by the data receiving unit.

Wherein the control unit stores the data in the storage unit when a network failure occurs between the data collecting apparatus and the passing instrument, and when the network fault between the data collecting apparatus and the passing instrument is recovered, To be transmitted.

And a status information transmission unit for transmitting the status information of the data collection device to the data collection management device.

The status information transmission unit may further transmit the status information to another data collection device.

The status information may include at least one of a data reception amount, a communication status, and storage space information per reference time.

According to another aspect of the present invention, there is provided a data collection management device comprising: a status information receiver for receiving status information from a data collection device collecting and transmitting data from a collection target; .

The information providing unit may provide an interface through which the user can select an element of the chart.

The element may include at least one of a selection of the data collection device, a data reception amount per reference time, a communication state, and storage space information.

The status information receiving unit may receive status information on the first data collecting apparatus among the plurality of data collecting apparatuses from the second data collecting apparatus.

And a setting unit configured to set a path through which the data collection device transmits the data.

The setting unit may provide an interface through which the user can design the path, and instruct the data collecting apparatus to operate according to the designed contents.

And a workflow setting unit for setting a flow of the process of collecting the data.

A data collection system according to another embodiment of the present invention includes a plurality of data collection devices for collecting data from a plurality of collection objects, a plurality of via devices for receiving the data from the plurality of collection devices and delivering the data to the destination device, And a data collection management device that receives status information from the plurality of data collection devices and displays the received status information on a chart, and provides the data to a user, wherein the plurality of data collection devices change paths to the plurality of pass- And transmit the data.

The plurality of data collecting apparatuses may transmit the data by changing paths to the plurality of passing apparatuses according to a data load to be processed.

The plurality of data collecting apparatuses may transmit the data by changing paths to the plurality of passing instruments according to the characteristic value of the data.

The plurality of data collection devices may store the data when a network failure occurs, and may transmit the stored data when the network failure is restored.

The data collection management apparatus receives status information about the first data collection apparatus among the plurality of data collection apparatuses from the second data collection apparatus and provides an interface through which a user can design a data transmission path, Thereby instructing the data collection device to operate.

According to the present invention, it is possible to accurately and efficiently collect big data, which is atypical and large-volume data, from each object to be collected, and to transfer the object to a storage system that is a target device stably even when a physical problem of the object occurs. Also, it is possible to easily provide information on the status of the collection process to the user, and to allow the user to stably manage the entire collection process.

1 is a block diagram of an environment in which a data collection device operates in accordance with an embodiment of the present invention.
2 is a block diagram of a data acquisition system in accordance with one embodiment of the present invention.
3 is a block diagram of a data acquisition device in accordance with an embodiment of the present invention.
4 is a block diagram of a data collection management device according to an embodiment of the present invention.
5, 6, and 7 are each an example of a screen provided by the data collection management apparatus according to an embodiment of the present invention for providing information.
8 is an example of a screen provided by the data collection management apparatus according to an embodiment of the present invention for user setting.
9 is a flowchart of a data collection method according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In order to clearly illustrate the present invention, parts not related to the description are omitted, and similar parts are denoted by like reference characters throughout the specification.

Throughout the specification, when an element is referred to as "comprising ", it means that it can include other elements as well, without excluding other elements unless specifically stated otherwise. Also, the terms " part, "" module," and " module ", etc. in the specification mean a unit for processing at least one function or operation and may be implemented by hardware or software or a combination of hardware and software have.

A data collecting apparatus, a data collecting method, a data collecting managing apparatus, and a data managing system according to an embodiment of the present invention will be described in detail with reference to the drawings.

FIG. 1 is a block diagram of an environment in which a data collection device operates in accordance with one embodiment of the present invention, and FIG. 2 is a block diagram of a data collection system in accordance with an embodiment of the present invention.

Referring to FIG. 1, a plurality of data collection devices 110, 120, and 130 are connected to a plurality of collection targets 10, 20, and 30 via a network to receive data from collection targets 10, 20, and 30. In this case, the data may be so-called big data, which is unstructured data that is difficult to manage as compared with existing data, such as generation amount, generation cycle, and generation format, and may include text and images .

The objects to be collected 10, 20 and 30 are devices for measuring data, for example, a sensor for measuring power generated from a generator of a power plant and a post generated from a server operating a social network service (SNS) A device for outputting a file, and the like.

The plurality of data collecting devices 110, 120 and 130 are connected to the plurality of passing vehicles 210 and 220 through a network and the plurality of passing vehicles 210 and 220 are connected to the destination device 300 through a network . The plurality of diesel vehicles 210 and 220 receive data from the data collection devices 110, 120 and 130 and transmit the data to the destination device 300.

In this case, the destination apparatus 300 may be a storage system for storing data for data analysis, a Hadoop distributed file system (HDFS), a central processing storage system, a file or a database. An analysis of the data stored in the destination apparatus 300 may be performed.

Each of the plurality of data acquisition devices 110, 120, and 130 forms an optional connection path with a plurality of the diesel engines 210 and 220. That is, the data collecting apparatus 110 forms paths P1 and P2 respectively with the passing apparatuses 210 and 220. The data collecting apparatus 120 is connected to the passing apparatuses 210 and 220 and paths P3 and P4, And the data collecting apparatus 130 forms paths P5 and P6 with the diesel engines 210 and 220, respectively. A plurality of diesel oil P1-P6 may be selected according to the conditions set by the user and data may be transmitted from the collecting apparatuses 110, 120 and 130 to the diesel engines 210 and 220.

Referring to FIG. 2, the data collecting apparatuses 110, 120 and 130 are connected to each other via a network. That is, the data collecting apparatuses 110 and 120 are connected to each other, the collecting apparatuses 120 and 130 are connected to each other, and the data collecting apparatuses 110 and 130 are connected to each other. As such, all data collection devices may be coupled to at least one of the collection devices other than (110, 120, 130) themselves.

The data collection devices 110, 120, and 130 are connected to the collection management device 400 via a network. The collection management device 400 is a server that receives the status information from the data collection devices 110, 120, and 130 to collect data from each of the collection targets 10, 20, and 30 and provides the collected status information to the user. The collection management device 400 may also set up a workflow associated with the operation of the data collection device 110, 120, 130 and apply it to the system. The collection management device 400 will be described in detail again.

The data collecting apparatuses 110, 120, and 130 will now be described in detail with reference to FIG.

3 is a block diagram of a data acquisition device in accordance with an embodiment of the present invention.

3, the data collecting apparatus 100 includes a data receiving unit 151, a storing unit 152, a data transmitting unit 153, a control unit 154, and a status information transmitting unit 155. [

The data receiving unit 151 receives data from the collection targets 10, 20, and 30.

The storage unit 152 stores the data received by the data receiving unit 151. The storage unit 152 may be implemented as a permanent or volatile storage device such as an optical disk such as a hard disk, a blu-ray disk, a flash memory, a database, various types of RAM, and the like. The storage unit 152 may be a device implemented outside the data collection device 100 or a storage place implemented on the web.

The data transferring unit 153 transfers the data received by the data receiving unit 151 or the data stored in the storage unit 152 to the transit devices 210 and 220.

The control unit 154 controls the transfer operation of the data transfer unit 153 according to the set conditions. In the control unit 154, a condition for adjusting the transmission path is set according to the state of the data collecting apparatus 100. For example, the setting is made for the basic transmission path, the failure transmission path, the load transmission path, and the data value transmission path Can be.

The basic transmission path is basically a transmission path when there is no problem in the network between the data collecting apparatus 100 and the transit apparatuses 210 and 220, and can be selectively changed to another transmission path. For example, the basic transmission path of the data transmission apparatus 110 may be set to P1, and the basic transmission path of the data transmission apparatus 120 may be set to P4.

The failure transmission path is a path selected in place of the basic transmission path when a failure occurs in the network between the data collecting apparatus 100 and the transit apparatuses 210 and 220. For example, when the failure transmission path of the data transmission apparatus 110 is set to P2 and the failure of the basic transmission path P1 occurs, the data transmission apparatus 110 can transmit data using P2. When the failure transmission path of the data transmission apparatus 120 is set to P3 and the failure occurs in the basic transmission path P4, the data transmission apparatus 120 can transmit data using P3.

The load transmission path is a path selected in place of the basic transmission path for load control when the transmission load between the data collecting apparatus 100 and the transit apparatuses 210 and 220 is larger than a predetermined reference. For example, if the amount of data transferred by the data collecting apparatus 110 using P1 is larger than the reference load, P2 may be set to be transmitted.

The data value transmission path is a path that is set differently according to the characteristics of data transmitted by the data collecting apparatus 100. For example, data having a header value A among the data transmitted by the data collecting apparatus 110 may be transmitted using the path P1, and data having a header value B may be configured to be transmitted using the path P2.

On the other hand, the control unit 154 may delay the transmission and store the data in the storage unit 152 according to the network status between the data collecting apparatus 100 and the passing vehicles 210 and 220. That is, when there is no problem in the network between the data collecting apparatus 100 and the transit apparatuses 210 and 220, the data receiving unit 151 transmits the received data according to the set route. When it is determined that there is a problem in the network situation between the data collecting apparatus 100 and the passing vehicles 210 and 220, the data receiving unit 151 stores the received data in the storing unit 152, The data stored in the storage unit 152 can be transmitted.

The status information transmitting unit 155 transmits information on the operation status of the data collecting apparatus 100 to the data collecting and managing apparatus 400. At this time, the status information of the data collecting apparatus 110 includes a data receiving amount per reference time, communication status information of the data collecting apparatus 100, and current storage space information of the storing unit 152.

Meanwhile, the status information transmitting unit 155 transmits status information to the other data collecting apparatuses 100 connected to the collecting and managing apparatus 400 as well as itself. That is, the data collection device 110 may transmit its status information to the data collection devices 120 and 130. Accordingly, the data collecting apparatuses 120 and 130 can transmit the status information of the data collecting apparatuses 110 other than their own status information to the data collecting and managing apparatus 400. At this time, the network between the data collection device 110 and the data collection management device 400 may not be smooth.

Meanwhile, the data collecting apparatus 100 may collect data and perform real-time analysis on the collected data.

Now, the data collection management device will be described in detail with reference to FIG.

4 is a block diagram of a data collection management device according to an embodiment of the present invention.

Referring to FIG. 4, the data collection management device 400 includes a monitoring unit 410 and a workflow setting unit 420.

The monitoring unit 410 receives status information from the data collecting apparatus 100 and provides the status information to the user and includes a status information receiving unit 411, an information providing unit 415, and a setting unit 417.

The status information receiving unit 411 receives the status information of the data collecting apparatus 100 from the data collecting apparatus 100. As described above, the status information receiving unit 411 may receive status information from each of the data collecting apparatuses 110, 120, and 130, but may receive data from other data collecting apparatuses 120 and 130 And may receive the status information of the collection device 110. [

The information providing unit 415 provides the status information received by the status information receiving unit 411 to the user. At this time, the information providing unit 415 may generate and provide status information on a chart according to a condition desired by the user. Accordingly, the information providing unit 415 may provide an interface to allow the user to set an information providing condition in order to generate a chart of a desired format. This will be described in detail with reference to FIG. 5 and FIG.

5, 6, and 7 are each an example of a screen provided by the data collection management apparatus according to an embodiment of the present invention for providing information. The user can access the web server by using his / her terminal (not shown).

5, the user can select a chart type 51 with a pie chart or a line chart, select a data collection device 100 to be provided 52, and select an element 53 to be included in the chart You can choose. Element 53 may be a data header, time, memory, and status of the data collection device.

Accordingly, state information for the data acquisition apparatus 100 can be generated by a pie chart or a line chart as shown in FIGS. 6 and 7.

4, the setting unit 417 establishes a data path between the data collecting apparatuses 110, 120, and 130, the passing vehicles 210 and 220, and the destination apparatus 300. For this, the setting unit 417 provides an interface through which a user can set a data path. This will be described in detail with reference to FIG.

8 is a view illustrating an example of a screen provided by the data collection and management apparatus according to an embodiment of the present invention for user setting. As shown in FIG. 8, the user has a data collection apparatus 110, 120, 130, 220, a fault transmission path, a load transmission path, and a data value transmission path can be freely selected to design the data path. The setting unit 417 then transmits the contents set by the user through the interface to the data collecting apparatuses 110, 120, and 130, the passing apparatuses 210 and 220, and the destination apparatus 300, . By this means, the user can design and manage the entire system of data collection as desired through the interface provided by the setting unit 417.

Referring again to FIG. 4, the workflow setting unit 420 may set a work flow related to the data collection system, and may transmit information on the determined workflow to each node so that the workflow may operate accordingly. At this time, each node refers to all subjects operating in the data collection system including the data collection device 100 and the pass-through devices 210 and 220.

The monitoring unit 410 and the workflow setting unit 420 may be physically separated from each other.

Now, a data collection method according to another embodiment of the present invention will be described in detail with reference to FIG.

9 is a flowchart of a data collection method according to an embodiment of the present invention.

Referring to FIG. 9, the data collecting apparatus 100 receives data from the collection targets 10, 20, and 30 (S910).

Then, the data collecting apparatus 100 determines whether a fault transmission path is set (S920). If it is determined in step S920 that there is a failure transmission path, it is determined whether there is a network failure (S930). If it is determined in step S930 that there is no network failure, data is transmitted using the basic transmission path (S940). If it is determined in step S930 that there is a network failure, the failure is transmitted using the failure transmission path (S950).

If it is determined in step S920 that the failure transmission path is not set, it is determined whether a load transmission path is set (S960). If it is determined in step S960 that the load transmission path is not set, the data is transmitted using the basic transmission path (S970). If it is determined in step S960 that the load transmission path has been set, the data is transmitted using the load transmission path (S980).

On the other hand, it is determined whether or not the data transmission is completed after the steps S940, S950, S970, and S980 (S990). If it is determined in step S990 that the data transmission is not completed, the data to be transmitted is stored (S991). Then, after the network failure or the like is restored, the restored data is transmitted (S992).

On the other hand, if the data value transmission path is set in steps S940, S950, S970, S980, S992, data can be transmitted through the path set according to the characteristic value of the data.

Although not shown in FIG. 9, status information may be transmitted to the data collection and management apparatus 400 sequentially or simultaneously with the procedure shown in FIG.

Although it is described in FIG. 9 that steps (S920-S992) are sequentially performed, each step is not limited to this order, and may be performed selectively or independently according to the situation.

The embodiments of the present invention described above are not implemented only by the apparatus and method, but may be implemented through a program for realizing the function corresponding to the configuration of the embodiment of the present invention or a recording medium on which the program is recorded. Such a recording medium can be executed not only on a server but also on a user terminal.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, It belongs to the scope of right.

Claims (22)

A data receiving unit for receiving data from a collection object,
A data transmission unit for transmitting the received data to the transit device, and
A control unit for controlling the operation of the data transfer unit according to the set transfer path,
Lt; / RTI >
Wherein the transmission path includes a basic transmission path and a load transmission path,
The control unit controls the data transmission unit to operate according to a basic transmission path,
And controls the data transfer unit to operate according to the load transfer path if the load amount is larger than the load amount reference set in the data collection device
Data acquisition device. delete The method of claim 1,
Wherein the transmission path includes a failure transmission path,
When a network failure occurs between the data acquisition device and the transit device, the control unit controls the data transfer unit to operate according to the failure transmission path
Data acquisition device. delete A data receiving unit for receiving data from a collection object,
A data transmission unit for transmitting the received data to the transit device, and
A control unit for controlling the operation of the data transfer unit according to the set transfer path,
Lt; / RTI >
Wherein the transmission path includes a data value transmission path that is differently set according to a basic transmission path and a characteristic value of the data,
The control unit controls the data transmission unit to operate according to a basic transmission path,
And controls the data transfer unit to operate according to the data value transmission path
Data acquisition device. A data receiving unit for receiving data from a collection object,
A data transfer unit for transferring the received data to a pass-through device,
A control unit for controlling the operation of the data transfer unit according to the set transfer path, and
A storage unit for storing the data received by the data receiver,
Lt; / RTI >
Wherein,
Storing the data in the storage unit when a network fault occurs between the data collecting apparatus and the passing instrument,
And controlling to transmit data stored in the storage unit when a network failure between the data collecting apparatus and the passing instrument is recovered
Data acquisition device. delete The method according to claim 1 or 5,
A status information transmitting unit for transmitting status information on the operation of the data collecting apparatus to the data collecting and managing apparatus,
Further comprising: 9. The method of claim 8,
Wherein the status information transmission unit further transmits the status information to another data collection apparatus. 9. The method of claim 8,
Wherein the status information includes at least one of a data reception amount, a communication status, and storage space information per reference time. A status information receiving unit for receiving status information from a data collecting device which collects and transmits data from a collection object, and
The state information is represented by a chart and is provided to a user.
Lt; / RTI >
Wherein the status information receiving unit receives status information on the first data collecting apparatus among the plurality of data collecting apparatuses from the second data collecting apparatus
Data collection management device. 12. The method of claim 11,
Wherein the information providing unit provides an interface through which the user can select an element of the chart
Data collection management device. The method of claim 12,
Wherein the element comprises at least one of a selection of the data collection device, a data reception amount per reference time, a communication status, and storage space information. delete 12. The method of claim 11,
And a setting unit for setting a path through which the data collection device transmits the data,
Further comprising: 16. The method of claim 15,
Wherein,
Providing a user with an interface for designing the path, and instructing the data acquisition device to operate according to the designed contents
Data collection management device. 12. The method of claim 11,
A workflow setting unit for setting a flow of the process of collecting the data;
Further comprising: A plurality of data collecting apparatuses each collecting data from a plurality of collection targets,
A plurality of diesel engines for receiving the data from the plurality of collection devices and delivering the data to the destination device, and
A data collection management device for receiving status information from the plurality of data collection devices, displaying the status information in a chart,
Lt; / RTI >
Wherein the plurality of data collecting apparatuses change the path to the plurality of passing apparatuses according to a data load to be processed and transmit the data. delete A plurality of data collecting apparatuses each collecting data from a plurality of collection targets,
A plurality of diesel engines for receiving the data from the plurality of collection devices and delivering the data to the destination device, and
A data collection management device for receiving status information from the plurality of data collection devices, displaying the status information in a chart,
Lt; / RTI >
Wherein said plurality of data collecting apparatuses change paths to said plurality of passing apparatuses according to the characteristic value of said data to transmit said data. A plurality of data collecting apparatuses each collecting data from a plurality of collection targets,
A plurality of diesel engines for receiving the data from the plurality of collection devices and delivering the data to the destination device, and
A data collection management device for receiving status information from the plurality of data collection devices, displaying the status information in a chart,
Lt; / RTI >
Wherein the plurality of data collecting devices store the data when a network fault occurs and transmit the stored data when the network fault is recovered
Data acquisition system. 22. A method according to any one of claims 18, 20 and 21,
The data collection management apparatus receives status information on the first data collection apparatus among the plurality of data collection apparatuses from the second data collection apparatus,
Provides an interface through which a user can design a data transmission path, and instructs the data collection device to operate according to the designed contents
Data acquisition system.

Images