KR102032680B1 - Distributed Parallel Processing System for Processing Data Of Continuous Process In Real Time - Google Patents

Abstract

The distributed parallel processing system that processes data for continuous processes in real time according to an aspect of the present invention capable of processing large amounts of data generated in a continuous process in real time, the data generated in a continuous process in which a plurality of processes are connected. In a distributed parallel processing system that processes continuous process data for processing in real time, the process identifier of a process in which the collection data is generated is mapped to the collection data collected from the continuous process, and the collection data collected in different processes At least one real-time processing unit for arranging the mapped mapping data for linkage processing between the at least one; And a plurality of memory units in which the alignment data are distributed and stored.

Description

Distributed Parallel Processing System for Processing Data of Continuous Process In Real Time}

The present invention relates to factory data processing, and more particularly to real-time processing of large data for continuous processing.

Each process is related to each other such that a plurality of processes for producing a finished product using raw materials are performed in succession, and the outputs of each process are mixed with each other or the state of the output of a specific process is supplied to a subsequent process. The production method is called continuous process production method. The steel industry, energy industry, paper industry, or refinery industry are typical industries to which continuous process production is applied.

For example, the steel industry consists of a plurality of processes, such as a steel making process, a steel making process, a playing process, and a rolling process. The iron making process is a process of producing molten iron (melting iron). Iron ore is put into a blast furnace and the iron ore is melted by the heat from burning raw coal. The steelmaking process removes impurities from the molten water. The iron and molten iron are put together in the converter and blown with oxygen to remove impurities. The regeneration process is a process in which iron in a liquid state becomes a solid, and molten steel without impurities is injected into a mold and then cooled and solidified while passing through a continuous casting machine to be made of an intermediate material such as slab, bloom, or billet. The rolling process is a process of making iron from steel sheets or wire rods, and slabs, blooms, or billets produced in the casting process are passed between rolls to produce steel sheets by stretching or thinning them.

In the case of the continuous process production method, unlike the industry in which the single process production method is applied, since the raw materials or intermediate materials move at a high speed, the data collection cycle is short and the data volume is large, and noise, dust, moisture Since products are produced in many factory environments, there are frequent measurement errors, and intermediate materials are mixed with each other or the location of materials moves depending on the method of work.

Accordingly, in an industry in which a continuous process is applied to a production method, a system capable of processing a large amount of data in real time and analyzing an association between data generated by each process is required.

However, the general factory data processing system disclosed in the Republic of Korea Patent Publication No. 10-2015-0033847 (name of the invention: digital factory capacity management system reflecting real-time factory status, published on April 2, 2015) (for example, steel data Since the processing system is for processing and analyzing data generated in a single process, it is not possible to process large amounts of data generated in a continuous process in real time, and also to analyze correlations among data generated by each process. There is a problem that can not be.

The present invention is to solve the above problems, it is characterized in that it provides a distributed parallel processing system for processing in real time the data for the continuous process that can process a large amount of data generated in the continuous process in real time.

In another aspect, the present invention is to provide a distributed parallel processing system for processing data for continuous processing in real time that can equally adjust the load of the real-time processing unit for the real-time processing of data.

In addition, another technical feature of the present invention is to provide a distributed parallel processing system that processes data for continuous processing in real time, which can improve the availability of a memory unit in which processed data is stored.

In order to achieve the above object, the distributed parallel processing system for processing data for continuous processing in accordance with an aspect of the present invention in real time, the continuous process data for processing data generated in the continuous process is connected to a plurality of processes In the distributed parallel processing system for processing in real time, the process identifier of the process in which the collection data is generated to the collected data collected from the continuous process, and the mapped for the cooperative process between the collected data collected in different processes At least one real-time processing unit to align the mapping data; And a plurality of memory units in which the alignment data are distributed and stored.

According to the present invention, there is an effect that can process a large amount of data generated in a continuous process in real time.

In addition, according to the present invention, by determining the execution modules to be distributed to each real-time processing unit based on the system resource usage information resulting from the execution of the execution module distributed to each real-time processing unit, it is possible to equalize the load of all the real-time processing unit. There is an effect that can improve the system processing performance.

In addition, according to the present invention, by configuring the memory unit tripled into one master instance and two slave instances, even if the master instance is down, any one of the two slave instances is transferred to the master to operate as a master-slave redundant structure. This is possible to maximize the availability of the memory unit.

1 is a diagram illustrating a smart factory architecture including a distributed parallel processing system for processing data for continuous processing in real time according to an embodiment of the present invention.
2 is a block diagram illustrating a configuration of an interface system according to an embodiment of the present invention.
3 is a diagram illustrating a configuration of an interface system including a plurality of interface processing units and a plurality of queue storage units.
4 is a view showing in detail the configuration of a distributed parallel processing system according to an embodiment of the present invention.
5 is a diagram illustrating a configuration of a distributed parallel processing system including a plurality of real-time processing units and a plurality of memory units.
6 is a conceptual diagram illustrating a distributed parallel processing method of data mapping and sorting by way of example.
7 is a diagram illustrating a configuration of a tripled memory unit.
8 is a view showing in detail the configuration of a big data analysis system according to an embodiment of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

The meaning of the terms described herein will be understood as follows.

Singular expressions should be understood to include plural expressions unless the context clearly indicates otherwise, and the terms “first”, “second”, etc. are used to distinguish one component from another. The scope of the rights shall not be limited by these terms.

It is to be understood that the term "comprises" or "having" does not preclude the existence or addition of one or more other features or numbers, steps, operations, components, parts or combinations thereof.

The term "at least one" should be understood to include all combinations which can be presented from one or more related items. For example, the meaning of "at least one of the first item, the second item, and the third item" means not only the first item, the second item, or the third item, but also two of the first item, the second item, and the third item, respectively. A combination of all items that can be presented from more than one.

1 is a diagram illustrating a smart factory architecture including a distributed parallel processing system for processing data for continuous processing in real time according to an embodiment of the present invention.

As shown in FIG. 1, the smart factory architecture according to the present invention is composed of layers such as a data collection device 1, a network 2, a smart factory platform 1000, and an application system 3.

The data collection device 1 collects data generated in a continuous process. In one embodiment, a continuous process is a method in which a plurality of processes for producing a finished product using raw materials are continuously performed, and the outputs of each process are mixed with each other or the state of the output of a specific process is changed and supplied to a subsequent process. Means the process of. The steel process is a representative example of such a continuous process. Hereinafter, for convenience of description, it will be described on the assumption that the continuous process is a steel process.

Since the steel process is composed of various processes such as a steel making process, a steel making process, a playing process, and a rolling process, the data collection device 10 is in the course of various processes such as a steel making process, a steel making process, a playing process, and a rolling process. Collect micro data generated. Here, the micro data means raw data as raw data itself collected through various sensors. Hereinafter, for convenience of description, micro data will be referred to as collected data.

To this end, the data collection device 1 includes various instruments, sensors, actuators and the like for collecting micro data. The data collection device 1 may further include a P / C, a programmable logic controller (PLC), a distributed control system (DCS), and the like, which integrate or control data collected by a measuring instrument, a sensor, an actuator, and the like.

The network 2 delivers a large amount of data collected by the data collection device 1 to the smart factory platform 1000. In one embodiment, the network 2 according to the present invention may include, but is not limited to, a network cable, a gateway, a router, or a wireless access point (AP).

The smart factory platform 1000 receives a large amount of micro data collected by the data collection device 1 through the network 2. The smart factory platform 1000 processes the received large amount of micro data in real time. In addition, the smart factory platform 100 not only determines in real time the presence of abnormalities such as equipment or materials based on the processed micro data, but also stores the processed micro data in the big data storage unit (not shown) for big data analysis. It provides search and analysis service for stored data.

In one embodiment, the smart factory platform 100 according to the present invention, as shown in Figure 1, the interface system 100, distributed parallel processing system 200, big data analysis system 300. In addition, the smart factory platform 1000 may further include a service system 400, a management system 500, and a security system 500.

The interface system 100 provides a connection means for connecting with heterogeneous devices of Level 0 to Level 2 through various protocols and normalizes the micro data by preprocessing the micro data collected by the data collection device 1.

Hereinafter, the interface system 100 according to the present invention will be described in detail with reference to FIG. 2.

2 is a diagram illustrating a configuration of an interface system 100 according to an embodiment of the present invention. As shown in FIG. 2, the interface system 100 according to the present invention includes an interface processing unit 110 and a queue storage unit 120.

The interface processing unit 110 preprocesses the collected data for linkage processing of the collected data collected from the continuous process. In one embodiment, the interface processing unit 110 may preprocess the collected data by standardizing the collected data. To this end, the interface processing unit 110 includes a receiving unit 111, a parsing unit 112, a standardization unit 113, a filtering unit 114, and a transmission unit 115, as shown in FIG. do.

The receiving unit 111 receives the collection data collected by the sensors included in the data collection device 1. In one embodiment, the receiving unit 110 may receive the collected data using one or more communication schemes. For example, the receiving unit 111 may use various communication methods such as iBA, OLE for Process Control (TCP), and the like. That is, since the communication method used by each gateway constituting the network 2 may vary according to the type of sensor or the like included in the data collection device 1, the receiving unit 111 according to the embodiment of the present invention It supports all communication methods that can communicate with various networks (2).

The parsing unit 112 parses the collected data received through the receiving unit 111.

In more detail, the collected data collected by the data collection device 1 may have a structure in which a group ID including a plurality of item IDs, a collection time, and a plurality of measured values are repeated. In this case, the item ID is used to identify the measured property and indicates a property of a facility, a material, or a product during the continuous process, and may be a temperature or humidity. The group ID may be a representative value of grouping several items by location or by each process in a specific factory. However, in the embodiment of the present invention, since the data structure is not limited thereto, the collected time may be included in the group ID itself.

As described above, when the collected data is configured in such a manner that a group ID, a collection time, and a plurality of measured values are repeated without any distinction, the distributed parallel processing system 200 does not have a standardized group ID, item ID, and Since a process of interpreting a plurality of measured values is required separately, real-time analysis of an abnormality of a facility or an abnormality of a product may be difficult.

Accordingly, the parsing unit 112 parses the collected data into meaningful units based on a preset layout for the cooperative processing of the collected data collected from the continuous process.

In one embodiment, the parsing unit 112 parses the collected data for each group ID, and matches a plurality of item IDs and a plurality of measurement values included in the group ID, respectively, to determine a single item ID, collection time, and a single measurement value. Each can be converted into data having the form

According to this embodiment, the parsing unit 112 may parse the collected data with reference to the message layout storage 117 in which the message layout for the collected data text is defined. The message layout storage unit 117 may be included in the interface processing unit 110 or another device in a separate configuration. However, since the present invention is not limited thereto, the parsing unit 112 may include information regarding the message layout.

The standardization unit 113 normalizes the collected data parsed by the parsing unit 112. In one embodiment, the standardization unit 113, for each of the collected data having a single item ID, the collected time, and a single measurement value delivered from the parsing unit 112, item ID according to a preset standard conversion criteria Parsed data can be standardized by converting to a standard ID ID and unifying the units and digits of the measurement.

Specifically, each sensor or actuator included in the data collection device 1 may have different item IDs according to the characteristics of the company in which they are produced or the factory in which they are included even if they measure the same property. As such, when data parsed based on different item IDs are transferred to the distributed parallel processing system 200 as they are, the process of interpreting non-standardized item IDs and a plurality of measured values is required separately. Alternatively, the performance of the distributed parallel processing system 200 for monitoring an abnormality of a product or the like may be deteriorated. Therefore, the standardization unit 113 according to the embodiment of the present invention may change the item ID included in each collection data into the standard item ID so that the data measuring the same property have the same item ID.

In this way, the standardization unit 113 preprocesses the collection data to have the same standard item ID with respect to the measured values measuring the same attributes, so that the collected data collected from the continuous process based on each standard item ID may be cooperatively processed. To help.

In addition, the format of the measurement value included in the collected data collected through the data collection device 1 may be different depending on the type of data collection device 1 such as a sensor or an actuator. As a result, a large amount of data collected from the continuous process is required for the process of converting the unit and the length of the data according to the type of each device in order to process the collected data of different formats collected from the continuous process. It is impossible to link the data in real time. Therefore, the standardization unit 113 according to the embodiment of the present invention standardizes the collected data so that the distributed parallel processing system 200 can process a large amount of data in real time.

The standardization unit 113 may standardize the item ID, the unit of measure and the number of digits by referring to the standard conversion reference storage unit 118 included in the interface processing unit 110 or another device in a separate configuration. Since the present invention is not limited thereto, the standardization unit 113 may include information on standard conversion criteria for standardization.

The filtering unit 114 determines whether to store the data normalized by the standardization unit 113 in the queue storage unit 120 according to a preset filtering criterion. For example, the grade is set in advance according to the collected type of collected data, and the filtering unit 114 may determine whether to store the data in the queue storage unit 120 according to the grade. In one embodiment, the grade may be determined based on the importance based on the standard item ID of the standardized data, but is not limited thereto.

The filtering unit 114 may filter the standardized data with reference to the filtering criteria storage unit 119 in which the criteria for filtering the data that need to be stored in the queue storage unit 120 are stored. The filtering criteria storage unit 119 may be included in the interface processing unit 110 or another device in a separate configuration. However, since the present invention is not limited thereto, the information about the filtering criteria may be stored in the filtering unit 114.

The transmission unit 115 stores the data filtered through the filtering unit 114 in the queue of the queue storage unit 120. In one embodiment, the transmission unit 115 may store the filtered data in the queue 121 of the queue storage unit 120 for each group ID or standard item ID.

That is, in the exemplary embodiment of the present invention, the collected data for each process having different formats are parsed and standardized and converted into a predetermined format, while the standardized data is stored for each group ID or standard item ID and stored for each group ID or standard item ID. Data can be checked, so collected data collected from continuous processes can be linked in real time.

In one embodiment, the queue storage unit 120 may be configured in plurality. According to this embodiment, if the transmission unit 115 stores the preprocessed collection data in any one of the plurality of queue storage unit 120 storage unit 120, the same data is also stored in the remaining queue storage unit 120 Duplicated and stored. In this case, the transfer unit 115 may store the collected data in the queue storage unit 121 having a low load in consideration of the load of the plurality of queue storage units 121.

In addition, the transmission unit 115 may determine whether to store the collected data according to the operation mode of the plurality of queue storage unit 120. In detail, the transmission unit 115 may stop storing data when the operation mode of the plurality of queue storage units 120 is a standby mode. In this case, an operation mode of the plurality of queue storage units 120 may be determined based on the number of the queue storage units 120 operating normally among the plurality of queue storage units 120.

On the other hand, when the operation mode of the plurality of queue storage unit 120 is the standby mode, the receiving unit 111 also stops receiving the collection data. That is, although the queue storage unit 120 operates abnormally and the collection data is not stored in real time, when the receiving unit 111 continuously receives the collection data from the data collection device 1, the queue is stored. Due to the failure of the unit 120, the collection data processing operation of the interface processing unit 110 also occurs.

Therefore, in the embodiment of the present invention, when the operation mode of the queue storage unit 120 is the standby mode, the interface processing unit 110 stops the reception and storage of the collected data to prevent the occurrence of a failure, and the queue storage unit 120 If) is normal, you can receive and store the collected data again.

The interface processing unit 110 according to an embodiment of the present invention may include at least one of the collection data merging unit 116, the message layout storage unit 117, the standard conversion criteria storage unit 118, and the filtering criteria storage unit 119. It may further include.

The collected data merging unit 116 merges the collected data and delivers the collected data to the parsing unit 112 in order to improve processing performance in a continuous process. In one embodiment, the collection data merging unit 116 merges the collection data received through the receiving unit 111 at a predetermined time interval (for example, 0.1 seconds, 1 second, 1 minute, etc.). That is, due to the nature of the continuous process, the collected data may be delivered to the parsing unit 112 in a very short period (for example, 5 ms to 20 ms). In this short cycle, the parsing unit 112 continuously parses the data. It can worsen the system's real-time processing performance.

Therefore, the collection data merging unit 116 is transferred directly to the parsing unit 112 without merging the collection data necessary for real-time monitoring, and the remaining collection data are merged at a predetermined time interval to generate a file form and then parsed unit ( 112). At this time, whether or not the collection data required for real-time monitoring may be set according to the importance of the collected data, for example, the collection data required for real-time monitoring of the collection data collected from the facility or material that needs immediate action in case of abnormality Can be set to

The message layout storage unit 117 stores data defining a layout for the entire collection data, and the parsing unit 112 refers to the message layout storage unit 117 for collection to be transmitted in binary form or as a separate data transmission standard. It can be used as a basis for interpreting data.

The standard conversion criterion storage unit 118 stores standard item IDs that standardize item IDs of various sensors constituting the data collection device 1, and reference units and digits corresponding to each standard item ID. That is, it is preferable that the collection data having the unit and the number of units standardized to the item ID and the standardized item ID are collected from the time when a large amount of the collection data is collected from the various sensors, but according to the characteristics of each process, Since the characteristics may be different, the embodiment of the present invention stores standard item IDs in which item IDs of various sensors and the like are standardized in advance, and reference units and digits according to each standard item ID, in order to effectively utilize them in future analysis.

According to such an embodiment, the above-described standardization unit 113 may change the item ID of the parsed data into the standard item ID by referring to the standard conversion criterion storage unit 118 and unify the unit and the number of digits.

The filtering criteria storage unit 119 stores criteria for filtering data that needs to be stored in the queue storage unit 120 among the standardized data, and the above-described filtering unit 114 stores the filtering criteria storage unit 119. The data to be stored in the queue storage unit 120 may be filtered among standardized data.

The queue storage unit 120 includes a queue 121 as an area for temporarily storing data preprocessed by the interface processing unit 110 before real-time processing.

The queue 121 is a storage for storing data preprocessed by the interface processing unit 110 for a predetermined time, and may store data based on a disk rather than a memory to prevent data loss. Spaces for storing data in the plurality of queues 121 may be divided into topics, and multiple partitions within the same topic may be separated and processed in parallel.

In one embodiment, a unique group ID may be assigned to each data group that the distributed parallel processing system 200 fetches from the queue storage unit 120, and a data fetch address may be managed for each unique group ID. The data may be stored and provided in the form of a queue that reads and writes data sequentially.

Although the above-described embodiment has been described as preprocessing the collection data through one interface processing unit 110 and one queue storage unit 120, in the modified embodiment, as shown in FIG. The collected data may be preprocessed through the processing unit 110 and the plurality of queue storage units 120.

According to this embodiment, the plurality of interface processing unit 110 may be extended in the form of adding according to the size of the data acquisition device 1 and the physical location of the factory, each interface processing unit 110 It may be implemented in a redundant structure for high availability (HA).

That is, each of the interface processing unit 110 is provided with an operation server and a backup server, the operation of the operating server in normal operation, when the failure of the operation server automatically activates the backup server, the operation of the interface processing unit 110 described above It can be performed continuously without interruption.

In addition, since the plurality of queue storage units 120 are implemented in a clustering structure, when data is stored in one queue storage unit 120, the data is replicated to another queue storage unit 120 to any queue storage unit 120. Even when a failure occurs, the service can be continuously provided with reference to another queue storage unit 120.

In addition, the interface processing unit 110 selects one queue storage unit 120 among the plurality of queue storage units 120 and stores the normalized data when standardization of the collected collected data is completed. In this case, a criterion for selecting the queue storage unit 120 to store data may be selected from various rules. For example, the queue storage unit 120 having the lowest load may be selected, or a method of sequentially selecting the data or data may be selected. It is possible to previously store and select the queue storage unit 120 to be stored for each sensor collected in advance.

Referring back to FIG. 1, the distributed parallel processing system 200 maps process identifiers to standardized data standardized by the interface apparatus 100, and maps data of each process for cooperative processing of collected data collected for each process. Sort them. Hereinafter, the distributed parallel processing system 200 according to the present invention will be described in more detail with reference to FIG. 4.

4 is a diagram illustrating a configuration of a distributed parallel processing system 200 according to an embodiment of the present invention. As shown in FIG. 4, the distributed parallel processing system 200 according to the present invention includes a real time processing unit 210 and a memory unit 220.

The real-time processing unit 210 generates mapping data by mapping process identifiers to data standardized by the interface system 100, and arranges mapping data so that linkage analysis of data such as operation, equipment, and quality may be performed.

To this end, the real-time processing unit 210 according to the present invention, as shown in Figure 4, the fetch execution module 211, the loading execution module 212, the process mapping execution module 213, performing data correction Module 215 and data sorting module 216. In addition, the real-time processing unit 210 may further include a facility abnormality detection execution module 217 and the quality abnormality detection execution module 218.

In one embodiment, the plurality of execution modules 211 ˜ 218 illustrated in FIG. 4 may be distributed to the real time processing unit 210 and implemented as an application for implementing each function, and these applications may be implemented in real time. After creating a work space in the processing unit 210 to create a plurality of threads (Thread) to perform the functions assigned to each execution module (211 ~ 218).

Hereinafter, functions of each of the plurality of execution modules 211 to 218 will be described in detail.

The fetch performing module 211 reads standardized data from the queue 121 of the interface system 100 and stores the standardized data in the collected data storage unit 221 of the memory unit 220. According to an exemplary embodiment, the fetch performing module 211 stores location information that previously inquired data for each of the plurality of queues 121 included in the queue storage unit 120, thereby storing the next data read before. Can read data

At this time, the interface unit 110 stores the standardized collection data in the queue 121 for each group ID or standard item ID for the cooperative processing of the collected data collected from the continuous process, the fetch performing module 211 is the queue 121 Standardized data stored in) can be read by group ID or standard item ID.

The loading execution module 212 loads the data stored in the collection data storage unit 221 and transmits the data to the process mapping execution module 213.

The process mapping performing module 213 maps a process identifier for identifying a process in which the data is performed to standardized data received from the loading performing module 212, and transfers the mapping data to the data correction performing module 215. .

In one embodiment, the process mapping performing module 213 may map at least one of a facility identifier of a facility performing each process or a material identifier of a material processed by the facility to standardized data as a process identifier. To this end, the mapping performing module 213 may include a facility mapping performing module 213a and a material mapping performing module 213b.

The facility mapping performing module 213a maps the facility identifier from which the data is collected to the standardized data loaded by the loading performing module 212. In one embodiment, the facility identifier may be a facility number assigned to each facility.

In an embodiment, the facility mapping performing module 213a may extract the facility identifier of the facility in which the collection data is generated based on the collection time of the collection data and the attribute information of the sensor that collected the collection data. In this case, the attribute information of the sensor may include the standard item ID as described above as information for identifying the measured attribute, but the present invention is not limited thereto.

That is, as described above, the data standardized by the interface device 100 includes a standard item ID, a collected time, and a single measurement value having standardized units and digits, which are operated for each operation performed by each facility. Since the time is determined, the facility mapping performing module 213a may check the equipment operated at the corresponding time based on the collection time information of the collected data. In addition, since the attribute information of the sensor includes equipment identifier information of the equipment measured by the corresponding sensor, the equipment mapping performing module 213a extracts the equipment identifier of the equipment measured by the sensor having a specific standard item ID at a specific time. Can be.

As a result, the mapping result data to which the equipment identifier is mapped includes the standard item ID, the collected time, the equipment identifier, and a single measurement value.

The material mapping performing module 213b maps the material identifier of the material processed through the facility corresponding to the facility identifier to mapping data to which the facility identifier is mapped. In one embodiment, the material identifier may be a material number assigned to each material. In the above-described embodiment, the material mapping module 213b has been described as mapping the material identifier to mapping data to which the equipment identifier is mapped. However, in the modified embodiment, the equipment identifier is pre-set to the data collected by the sensor. Since it may be included, the material mapping performing module 213b may map a material identifier to standardized data transmitted from the loading performing module 212.

The material mapping performing module 213b may extract the material identifier of the material processed in the facility corresponding to the facility identifier mapped to the mapping data based on the work instruction information performed in each process.

For example, when the first material to which the first material identifier is assigned is generated through the first facility of the first process, the material mapping module 213b may use the first material to map data to which the facility identifier of the first facility is mapped. Map additional identifiers. In addition, when the second material to which the second material identifier is assigned is generated through the second facility that performs the second process, the material mapping performing module 213b may be configured to map data to which the facility identifier of the second facility is mapped. Further map the second material identifier.

The data to which the equipment identifier is mapped by the equipment mapping performing module 213a includes a standard item ID, a collection time, a equipment identifier, and a single measured value having standardized units and digits, and thus is output from the material mapping performing module 213b. The mapping data to be included will include a standard item ID, time collected, equipment identifier, material identifier, and a single measurement.

As described above, the distributed parallel processing system 200 according to an embodiment of the present invention maps a process identifier including at least one of a facility identifier and a material identifier to standardized data, thereby allowing any material to pass through any facility for each data. It is possible to check whether the data collected in the process, and through the tracking of this data can be linked analysis between each process.

In addition, since the collected data is standardized through the interface system 100, the distributed parallel processing system 200 may map a facility identifier and a material identifier to standardized data having a standardized structure, thereby real-time large amount of data without time delay. Can be processed as

Meanwhile, the collection data collected through the data collection device 1 may include load data, which is data collected while the facility is processing materials, and no load data, which is data collected while the facility is not processing materials. Can be.

In the case of no-load data, since there is no material identifier to additionally map to the collection data to which the facility identifier is mapped, the material mapping performing unit 213b may store the mapping data to which the facility identifier is mapped directly in the alignment data storage unit 222. . In this case, the load data and the no-load data may be stored separately in the alignment data storage unit 222.

The data correction performing module 215 corrects the mapping data by adding the missing data when there is missing data among the mapping data to which the process identifier is mapped. In one embodiment, the data correction performing module 215 may match a collection time included in the mapping data to a predetermined collection period in order to correct the mapping data. That is, since the collected data should be generated at a predetermined collection period through the data collection device 1, there may be a case in which the data is not corrected. Therefore, the data correction performing module 215 matches the collection time of each mapping data to the collection period.

For example, when storing the collection data with a collection period of 20 ms, since the data collection device 1 should generate the collection data every 20 ms, the data correction performing module 215 has a collection time of 15:01:11 seconds 0005 ms. The collection time of mapping data may be changed to 15:01:11 seconds 0000 ms, and the collection time of mapping data having a collection time of 15:01:11 seconds 0050 ms may be changed to 15:01:11 seconds 0040 ms.

In this case, when the collection period is 20 ms, the collection data should be delivered in 20 ms. When some collection data is missing, the collection period in which each collection data is collected is longer than 20 ms. Accordingly, the data correction performing module 215 corrects the missing data by using mapping data at a position closest to the area where the data where the missing data should have been collected or mapping data at a collection time adjacent to the time when the missing data occurs.

The data sorting performing module 216 sorts the mapping data corrected by the data correcting performing module 215 or mapping data to which the process identifier is mapped for linkage processing between data of each process.

In one embodiment, the data sorting performing module 216 sequentially sorts mapping data mapped with the same material identifier according to a collection time. That is, the data sorting performing module 216 may sort the mapping data in the order of materials in units of materials having the same material identifier for linkage processing between collected data collected in a continuous process.

The data sorting performing module 216 sorts the mapping data arranged in chronological order based on the location at which the data is collected on the material corresponding to the same material identifier.

For sorting the mapping data, the data sorting performing module 216 collects the collection position of the collection data arranged over time on the material using at least one of the length of the material, the moving speed of the material, and the collection period of the collection data. Can be determined. For example, the data sorting performing module 216 may determine a collection position at which collection data is collected for each cycle on the material based on a product of a moving speed of the material and a collection cycle of the collection data and the total length of the material. Accordingly, mapping data arranged over time may be aligned with data measured at predetermined positions in one direction on the material.

The data sorting performing module 216 may perform a distance between reference points set at predetermined intervals on each material and a collection position of each mapping data for linkage processing of the collected data collected from the first process and the second process at different collection cycles, respectively. A measurement value at each reference point is calculated on the basis of, and reference data at each reference point is generated based on the calculated measurement value.

The data sorting performing module 216 sequentially sorts the collected data arranged in time and the reference data at the reference points in one direction on the material. The data sorting performing module 216 stores the sorted data in the sorting data storage unit 222 of the memory unit 220. In one embodiment, the one direction may be at least one of the longitudinal direction of the material, the width direction of the material, and the thickness direction of the material, but the present invention is not limited thereto.

In the first process and the second process, each collection data is collected at different collection cycles, and the length, width, or thickness of the material passing through the first process is different from the length, width, or thickness of the material passing through the second process. As such, it may be difficult to manage the association of the cross-sectional changes of the measured values measured in a particular region of the material in each process. Therefore, in the present invention, for the cooperative processing between the collected data collected from the first process and the second process, the measurement value at a plurality of reference points set on the material processed in each process is calculated, and the measurement at the reference point for each material The value is used to manage the mapping data between the processes.

Hereinafter, an example in which the data sorting module 216 aligns the reference data in the longitudinal direction on the material will be described in detail.

First reference points are set at predetermined intervals in the longitudinal direction of the first material processed in the first step, and second reference points are set at predetermined intervals in the longitudinal direction of the second material processed in the second step. In this case, a first material identifier corresponding to the first material is mapped to the first reference data at the first reference points, and a second material corresponding to the second material is mapped to the second reference data at the second reference points. Identifiers are mapped. Accordingly, the first reference data and the second reference data are linked based on the first material identifier and the second identifier on a material family tree (not shown) in which material identifiers are mapped for each material.

That is, the material family tree is connected in the form of each material layer tree, and the mapping data of each process is assigned through the material identifier assigned to the material generated by sequentially passing the first process and the second process by referring to the material family tree. Can be linked to each other.

The data sorting performing module 216 stores the mapping data and the reference data arranged in the longitudinal direction of the material in the alignment data storage unit 222 of the memory unit 220 as described above.

In one example, the data sorting performing module 216 may store the sorting data having material identifiers associated with each other based on the material family tree in the same space of the sorting data storage 222. As a result, when the collected data are linked, the linked collected data can be easily used in the same storage space.

In addition, the data sorting performing module 216 may sort the first sorting data in which the mapping data is sorted according to time, the second sorting data in which the mapping data is sorted based on the collection position, and the reference data at the reference points. 222 can be stored separately.

That is, the data sorting performing module 216 stores the first sorting data, the second sorting data, and the reference data in separate storage locations, respectively, so that the first sorting data and the second sorting data corresponding to the actually measured values are stored. The reference data corresponding to a virtual value calculated based on actual data may be separately used based on time and a collection position, but the present invention is not limited thereto.

In addition, the data sorting performing module 216 stores the event indicating the completion of the sorting in the complete event storage unit 223 so that the sorting result can be used in the large-capacity big data analysis system 300.

As such, the real-time processing unit 210 maps process identifiers, such as equipment identifiers or material identifiers, to the collected data, and arranges the mapping data so that the collected data can be cooperatively processed.

The facility abnormality detection performing module 217 receives data mapped with the facility identifier from the facility mapping performing module 213a and determines whether there is a facility abnormality according to a preset facility abnormality criterion. If it is determined that an abnormality has occurred in a specific facility, the facility abnormality detection performing module 217 stores the determination result in the abnormality detection result storage unit 224 of the memory unit 220.

At this time, if the collected data collected as an example of the facility abnormality determination criteria for determining whether or not the equipment abnormality lasts for a predetermined time out of a predetermined reference value, it may be set to determine that the abnormality in the equipment. The present invention is not limited to this.

The quality abnormality detection performing module 218 loads the alignment data from the alignment data storage unit 222 and determines whether there is a quality abnormality according to a preset quality abnormality determination criterion based on the alignment data. If it is determined that an abnormality has occurred in the quality of the specific material, the quality abnormality detection performing module 218 stores the determination result in the abnormality detection result storage unit 224 of the memory unit 220.

In one embodiment, the quality abnormality detection performing module 218 generates macro data to be used as a reference of the quality abnormality determination equation through operations such as average and error prediction of the collected data, and whether the collected data is abnormal or not. Input to the judgment formula can determine the quality abnormality according to the result.

As described above, the distributed parallel processing system 200 according to the present invention sorts the standardized data by material unit for linkage analysis between the processes, and is ideal for the quality of equipment or materials in real time based on collected data or sorted data. By monitoring whether this occurs, it is possible to predict the failure of the equipment in advance.

The real time processing unit 210 may further include a facility information storage unit 219a, a work instruction information storage unit 219b, a sensor attribute information storage unit 219c, and a quality determination model storage unit 219d. .

The facility information storage unit 219a includes information on which factories and which processes are located at which locations, maintenance history information about the facility, facility information to be mapped to data, and facilities for determining whether there is an abnormality of each facility. Equipment failure judgment criteria are stored.

The work instruction information storage unit 219b includes instruction information on the work of the Manufacturing Execution System (MES) system, material identifier information generated in a specific facility as the work, quality indicators for performing the work, and material information to be mapped to data. And quality judgment criteria for determining whether the quality of the material is abnormal for each material is stored.

The sensor attribute information storage unit 219c stores information such as the type, unit, collection cycle, corresponding equipment identifier, and factory / process of data collected from the sensor, and in particular, the item ID of the sensor measuring the equipment or material is stored. have.

Since the quality determination model storage unit 219d stores the equipment abnormality determination criteria and the quality abnormality determination criteria in advance, the quality abnormality detection performing module 218 may determine the quality abnormality with reference to the quality determination model storage unit 219d. have.

The memory unit 220 stores various data generated by the real-time processing unit 210, specifically, the collected data storage unit 221, the alignment data storage unit 222, the completion event storage unit 223, and the abnormality detection unit. Result storage unit 224.

The collected data storage unit 221 stores the standardized data read from the queue 121 through the fetch performing module 211, that is, the standardized data before mapping, correction, and sorting is performed for each standard item ID. The loading execution module 212 loads standardized data from the collection data storage unit 221 and transfers the standardized data to the facility mapping execution module 213a.

The alignment data storage unit 222 includes mapping data in which at least one of the equipment identifier and the material identifier are mapped by the equipment mapping performing module 213a and the material mapping performing module 213b so as to determine the quality of the equipment or material. Stored in a sorted state. In an embodiment, the alignment data storage unit 222 may separately store load data to which both the equipment identifier and the material identifier are mapped and no-load data to which only the equipment identifier is mapped.

Completion event storage unit 223 is stored according to the collection cycle and the data that is missing the correction data is stored in the alignment data storage unit 222, the sorting result is completed to use in the big data analysis system 300 Informing events are stored. Accordingly, the big data analysis system 300 monitors the completion event storage unit 223 and extracts the alignment data from the alignment data storage unit 222 when a new completion event occurs.

Specifically, the completion event includes a corresponding event transmission time, data collection time, key information for reading data from the alignment data storage unit 222, partition and directory information for storing an event, and the like. Therefore, when a new completion event is acquired in the completion event storage unit 223, the big data analysis system 300 uses the key information included in the completion event to display data corresponding to the completion event in the alignment data storage unit 222. Newly sorted data can be obtained by checking which partitions and directories are stored in).

The abnormality detection result storage unit 224 stores the abnormality detection result of the specific facility detected by the facility abnormality detection execution module 217 and the quality abnormality detection result of the specific material detected by the quality abnormality detection execution module 218.

Therefore, in the case of the present invention, the user accesses the abnormality detection result storage unit 224 through a separate abnormality monitoring system (not shown) built outside the smart factory platform 1000 of the smart factory platform 1000. You can check whether there is an abnormality in the quality of a particular plant or material.

However, the present invention is not limited thereto, and when an abnormality occurs in the quality of a specific facility or a specific material, the user may directly check the abnormality by directly transmitting the result to the abnormality detection monitoring system.

In the above-described embodiment, the distributed parallel processing system 200 has been described as mapping and sorting normalized data through one real-time processing unit 210 and one memory unit 220. However, in the modified embodiment, The distributed parallel processing system 200 may map and sort the standardized data using the plurality of real-time processing units 210a, 210b, and 210c and the plurality of memory units 220 as illustrated in FIGS. 4 and 5. have.

Hereinafter, a distributed parallel processing system according to a modified embodiment will be described with reference to FIGS. 4 and 5.

5 is a diagram schematically illustrating a configuration of a distributed parallel processing system including a plurality of real-time processing units and a plurality of memory units.

As shown in FIG. 5, the distributed parallel processing system 200 includes a plurality of real-time processing units 210a, 210b, and 210c, a plurality of memory units 220a, 220b, and 220c, and a performing module distribution unit 230. Include.

One or more execution modules 211 to 218 of the plurality of execution modules 211 to 218 for mapping and sorting standardized data are distributed to the plurality of real time processing units 210a, 210b, and 210c. As described above, the present invention executes the mapping and sorting of the standardized data by executing the execution modules 211 to 218 distributed to them by the plurality of real time processing units 210a, 210b, and 210c, respectively. The distributed parallel processing may be performed through 210b and 210c to prevent overload as all the execution modules 211 to 218 are performed in a single real-time processing unit.

That is, the plurality of real-time processing units 210a, 210b, 210c, and 210d may perform the fetch execution module 211, the loading execution module 212, the facility mapping execution module 213a, the material mapping execution module 213b, and data correction. At least one of the module 215, the data sorting performing module 216, the facility abnormality detecting performing module 217, and the quality abnormality detecting performing module 218 are distributed and processed in parallel, and the final result data is stored in memory. By storing in the unit 220, a large amount of data transmitted from the interface system 100 is processed in real time.

In this case, a plurality of execution modules 211 to 218 performing the same function may be distributed to one real-time processing unit 210a, 210b, 210c. For example, a plurality of facility mapping performing modules 213a may be disposed in one real-time processing unit 210a, 210b, or 210c.

In one embodiment, the plurality of real-time processing unit 210a, 210b, 210c may be configured in a clustering structure. As described above, since the plurality of real-time processing units 210a, 210b, and 210c have a clustering structure, when a failure occurs in a specific real-time processing unit, the execution modules 211 to 218 that are running in the real-time processing unit that has failed are transferred to another real-time processing unit. It can be moved to ensure availability.

In FIG. 5, the distributed parallel processing system 200 includes three real-time processing units 210a, 210b, and 210c, but the real-time processing unit may be additionally expanded according to data processing performance.

The plurality of memory units 220 stores data processed by the plurality of real-time processing units 210a, 210b, and 210c. In an embodiment, in order to increase processing performance and to ensure availability in the event of a failure, the plurality of memory units 220 may have a clustering structure like the above-described queue storage unit 120.

That is, when data is stored in one memory unit 220, the data is duplicated and stored in the other memory unit 220, so that even if a failure occurs in a specific memory unit 220, the service is continuously provided through the other memory unit 220. Can be provided.

In an embodiment, the plurality of memory units 220 may be provided in a redundant structure for high availability (HA). That is, each memory unit 220 includes a master instance (M) and a slave instance (S). In this case, the master instance M included in the first memory unit 220a and the slave instance S included in the second memory unit 220b operate as a pair and are included in the second memory unit 220b. The master instance M and the slave instance S included in the first memory unit 220a operate in pairs.

In this case, when the alignment data is stored in the master instance M of the first memory unit 220a, the alignment data is duplicated and stored in the slave instance S of the second memory unit 220b, and the second memory unit 220b is stored. When the alignment data is stored in the master instance (M), the alignment data is duplicated and stored in the slave instance S of the first memory unit 220a.

In one embodiment, the sorted data recorded in the slave instance (S) may be backed up to a file in the form of a scripter (Scripter) for recovering in case of failure. In this case, the script file refers to a file in which a command related to writing or reading data is stored together with the corresponding data.

Meanwhile, when the master instance M included in the first memory unit 220a operates and a failure occurs, the slave instance S included in the second memory unit 220a is automatically activated, thereby realizing the aforementioned real-time processing unit ( Various functions of 210 may be continuously implemented without interruption.

In one embodiment, the master instance (M) and the slave instance (S) of each memory unit 220 is configured in a single thread (Rhread) form, the instance and port are separated for each write and read.

On the other hand, the master instance (M) and the slave instance (S) included in each memory unit 220 is the above-described collection data storage unit 221, alignment data storage unit 222, completion event storage unit 223 and the above Each of the sensing result storage units 224 may be configured.

Hereinafter, referring to FIG. 6, a method of distributed parallel processing for mapping and sorting of standardized data will be described as an example.

As shown in FIG. 6, since the fetch performing module 211 is distributed to the first real time processing unit 210a, the first real time processing unit 210a executes the fetch performing module 211 to execute the fetch performing module. 211 accesses the queue 121 to fetch standardized data and stores the fetched data in the master instance M of the first memory unit 220a. At this time, data is also copied and stored in the slave instance S of the second memory unit 220b. In the above-described embodiment, it has been described that data is stored in the master instance M of the first memory unit 220a. However, data may be stored in the master instance M of the second memory unit 220b. In this case, data is copied and stored in the slave instance S of the first memory unit 220a.

Since the loading execution module 212, the facility mapping execution module 213a, and the material mapping execution module 213b are distributed in the second real time processing unit 210b, the second real time processing unit 210b may be the loading execution module. By executing 212, the loading execution module 212 reads data from the slave instance S of the second memory unit 220b or the slave instance S of the first memory unit 220a.

The second real-time processing unit 210b reads from the slave instance S of the second memory unit 220b or the slave instance S of the first memory unit 220a by executing the facility mapping performing module 213a. The equipment identifier is mapped to the data, and the material mapping function module 213b is executed to map the material identifier to the data to which the equipment identifier is mapped.

Since the data correction performing module 215 and the data alignment performing module 216 are distributed in the third real time processing unit 210c, the third real time processing unit 210c performs the mapping by executing the data correction performing module 215. By correcting the missing data among the data, and performing the data sorting module 216, the corrected mapping data is sorted by material unit and stored in the master instance M of the second memory unit 220b. At this time, data is also copied and stored in the slave instance S of the first memory unit 220a. In the above description, the alignment data is described as being stored in the master instance M of the second memory unit 220b. However, the alignment data may be stored in the master instance M of the first memory unit 220a. In this case, data is copied and stored in the slave instance S of the second memory unit 220b.

In the above-described embodiment, the plurality of memory units 220 are configured in a redundant manner so that the master instance M included in the first memory unit 220a and the slave instance S included in the second memory unit 220b are included. Has been described as a pair, and the master instance M included in the second memory unit 220b and the slave instance S included in the first memory unit 220a operate as a pair.

However, according to this embodiment, since the master instance M and the slave instance S are implemented as a single thread, when the master instance M of the first memory unit 220a is down, the first memory unit ( During the down time until the master instance M of the controller 220 is normalized, there is a limitation that the slave instance S of the second memory unit 220b cannot service both the write operation and the read operation.

Therefore, in the modified embodiment, the memory unit 220 may be implemented in a triple structure as shown in FIG. 7. Specifically, each of the memory units 220 according to the modified embodiment includes a master instance M, a first slave instance S1, and a second slave instance S2.

In this case, the master instance M included in the first memory unit 220a operates in pairs with the first slave instances S1 of the second and third memory units 220b and 220c. Accordingly, when data is written in the master instance M included in the first memory unit 220a, the data is copied and stored in the first slave instances S1 of the second and third memory units 220b and 220c. do.

In addition, the master instance M included in the second memory unit 220b includes the first slave instance S1 included in the first memory unit 220a and the second slave instance included in the third memory unit 220c. It works in pairs with (S2). Accordingly, when data is written to the master instance M included in the second memory unit 220b, the data is included in the first slave instance S1 and the third memory unit 220c included in the first memory unit 220a. Data is also replicated and stored in the second slave instances S2.

In addition, the master instance M included in the third memory unit 220c operates in pairs with the second slave instances S1 included in the first memory unit 220a and the second memory unit 220b. . Accordingly, when data is written in the master instance M included in the third memory unit 220c, the data is also written to the second slave instances S1 included in the first memory unit 220a and the second memory unit 220b. Data is duplicated and stored.

Referring back to FIG. 5, the execution module distribution unit 230 distributes the plurality of execution modules 211 to 218 to the plurality of real time processing units 210a to 210c. In addition, the execution module distribution unit 230 includes a plurality of execution modules according to the load of the real time processing units 210a to 210c due to the execution of the execution modules 211 to 218 distributed to the real time processing units 210a to 210c. Redistribution (211 ~ 218) to the real-time processing unit (210a ~ 210c).

In detail, the execution module distribution unit 230 includes a execution module storage 232, a distribution order determining unit 234, and a distribution execution unit 236.

First, the execution module storage 232 stores a plurality of execution modules 211 to 218 for performing mapping and sorting of standardized data.

The distribution order determiner 234 is configured to distribute the plurality of execution modules 211 to 218 to each of the real time processing units 210a to 210c by the distribution execution unit 236, and then to the real time processing units 210a to 210c. The order for redistribution of the execution modules 211 to 218 is determined based on the resource usage information.

Even though the same number of execution modules 211 to 2180 are distributed to each of the real-time processing units 210a to 210c, since the resource utilization rate is different for each execution module 211 to 218, the execution module 211 to 218 having a high resource usage rate is performed. Since the distributed real-time processing units 210a to 210c have a heavy load compared to the distributed real-time processing units 210a to 210c in which the resource utilization rate low performing modules 211 to 218 are distributed, distributed parallel processing systems. The performance of 200 is degraded.

Therefore, the distribution order determining unit 234 according to the present invention determines the resource usage information of the real-time processing unit (210a ~ 210c) by the execution of the execution module (211 ~ 2180) initially distributed, a plurality of real-time processing unit ( The distribution order for redistribution of the plurality of performance modules 211 to 218 may be determined so that the load of 210a to 210c may be adjusted. In one embodiment, the distribution order determiner 234 may determine the distribution order for redistribution of the plurality of execution modules 211 to 218 so that the load of the plurality of real-time processing units 210a to 210c is equalized. Here, the distribution order determination means determining the real-time processing units 210a to 210c to which each performing module 211 to 218 is distributed. At this time, the number of execution modules 211 to 218 to be distributed to each of the real-time processing units 210a to 210c may be equally set.

According to an embodiment, the distribution order determiner 234 distributes the plurality of execution modules 211 to 218 using at least one of an average value of system resources and a usage pattern of system resources according to execution of the performance module. Determine the order. According to such an embodiment, the system resource may include at least one of CPU utilization rate, memory usage amount, network communication amount, and disk input / output throughput of each real-time processing unit 210a to 210c.

The distribution execution unit 236 distributes and distributes the plurality of execution modules 211 to 218 to the plurality of real time processing units 210a to 210c according to the distribution order determined by the distribution order determiner 234.

Specifically, the distribution execution unit 236 initially distributes the plurality of execution modules 211 to 218 stored in the execution module storage 232 to the real time processing units 210a to 210c arbitrarily. Subsequently, when the predetermined pause period, the distribution execution unit 236 collects the execution modules 211 to 218 distributed to each of the real-time processing units 210a to 210c and stores them in the execution module storage 232. The plurality of execution modules 211 to 218 stored in the execution module storage 232 are distributed to the corresponding real-time processing units 210a to 210c according to the distribution order determined by the distribution order determining unit 234.

As described above, the present invention can efficiently distribute the plurality of execution modules 211 to 218 to the real time processing units 210a to 210c by the execution module distribution unit 230, and between the real time processing units 210a to 210c. It is possible to balance the load, thereby improving the overall performance of the distributed parallel processing system 200.

Referring back to FIG. 2, the big data analysis system 300 stores the data arranged by the distributed parallel processing system 200 in the big data storage space. In addition, the big data analysis system 300 manages to prevent data loss and provides an inquiry function for historical data. Hereinafter, the big data analysis system 300 according to the present invention will be described in detail with reference to FIG. 8.

8 is a block diagram showing the configuration of the big data analysis system shown in FIG.

As shown in FIG. 8, the big data analysis system 300 includes a large data processor 310, a big data storage 320, and a query processor 330.

The large-capacity data processing unit 310 distributes and parallelizes the alignment data and the abnormality detection result, and includes a completion event receiving unit 311, an alignment data fetch unit 312, a memory queue 313, a file generation unit 314, and An abnormality detection data receiving unit 315 is included.

The completion event receiving unit 311 monitors the completion event storage unit 223 included in the distributed parallel processing system 200 and transmits the completion event to the sort data fetch unit 312 when the completion event is newly stored.

When the completion event is transmitted from the completion event receiving unit 311, the alignment data fetch unit 312 inquires the alignment data corresponding to the completion event from the alignment data storage unit 222 and stores the alignment data in the memory queue 313. In one embodiment, the alignment data fetch unit 312 checks in which partition and directory of the alignment data storage unit 222 the data corresponding to the completion event is stored using key information included in the completion event. The data stored in the alignment data storage unit 222 may be inquired and stored in the memory queue 313.

The memory queue 313 temporarily stores the data in the memory before storing the data in the big data storage 320. In one embodiment, when the amount of data is not large and loss prevention is required, the memory queue 313 may be implemented on a file basis.

The file generation unit 314 generates data stored in the memory queue 313 as a physical file and stores the data in the big data storage unit 320. In an embodiment, the file generation unit 314 may change or compress the file format when storing the file in the big data storage unit 320.

The abnormality detection data receiving unit 315 monitors the abnormality detection result storage unit 224 included in the distributed parallel processing system 200 and stores the new abnormality detection result in the memory queue 313.

The big data storage unit 320 stores the file generated by the file generation unit 314. In one embodiment, the big data storage 320 may be implemented based on a distributed file system. For example, the big data storage 320 may be implemented as a Hadoop based distributed file system.

According to this embodiment, the big data storage 320 is composed of a master node 320a and a data node 320b. The master node 320a stores a large amount of files generated by the mass data processing apparatus 300 in the data nodes 320b and generates a job for querying the data stored in the data node 320b. And manages and manages metadata.

Here, the job Job refers to a unit for processing a query received from the query processing unit 330 to inquire data stored in the data node 320b. For example, when a query for querying data is executed for one table recorded in the data node 320b, if ten data nodes 320b are selected as the result of querying the data of the table, 10 The job Job which retrieves and retrieves data for each of the data nodes 320b and the job Job that integrates the data obtained for each data node 320b are executed with respect to the data nodes 320b.

Metadata includes a location of a file stored in the data node 320b, a file name, a block ID where the file is stored, a storage location of the server, and the like. For example, if a file is generated in the file generating unit 314, the file's location and file name are stored in metadata, and if the file is larger than the block size and divided into five blocks, the file is stored in three different servers. The ID and storage location of each server will be stored in addition to the metadata.

Such metadata is used as location information of data when distributing each job and loading data of a specific file when executing a job for inquiry of data stored in the data node 320b.

The data node 320b stores a large amount of files generated by the mass data processing apparatus 300. In one embodiment, a plurality of data nodes 320b may be implemented, and each data node 320b includes a historical data storage 322 and a model storage 324.

The historical data storage unit 322 included in each data node 320b stores not only the file generated by the file generating unit 314 but also a large amount of collected data collected in real time by the data collecting device 1. do. In one embodiment, the file generated by the file generation unit 314 may be stored in a separate relational database (RDB).

The model storage unit 324 stores a quality judgment model and an abnormality prediction model necessary for determining the quality of a material or a product through the service system 400.

The query processing unit 330 is configured to query the data stored in the big data storage 320 and return data. The query processing unit 330 includes a query receiving unit 332, a query execution unit 336, and a query result transmission unit 338. . The query processor 330 may further include a query scheduling unit 334.

Specifically, the query receiving unit 332 receives a query from the user and interprets the received query syntax.

The query execution unit 336 transmits the query received through the query receiving unit 322 to the big data store 320 to execute the query, and obtains a query execution result from the big data store 320.

The query result transmission unit 338 delivers the data obtained from the query execution result big data store 320 to the user who requested the query.

Meanwhile, when the query received through the query receiving unit 332 includes a plurality of subqueries, the query scheduling unit 334 classifies the received query into subqueries and delivers the received query to the query execution unit 336. do.

Referring back to FIG. 1, the service system 400 is a structure in which standardized processing processes and business standards are recycled as services, and plans-execution is performed through connection between services defined as functional units by repositories of business know-how. -Facilitate the linkage between controls and invoke analysis models that include quality judgment models or anomaly prediction models for materials or products, and proceed with the analysis results.

Since the analysis model is previously stored in the model storage unit 324 illustrated in FIG. 8, when the execution call event for the analysis model is input, the service system 400 extracts data necessary for the analysis model from the model storage unit 324. Provide results.

That is, the service system 400 directly receives and analyzes the alignment data processed by the distributed parallel processing system 200, or when the alignment data is stored in the big data storage 320 of the big data analysis system 300, The data may be analyzed with reference to the data storage 320.

The management system 500 manages configuration files for management of individual configurations belonging to the smart factory platform 1000 and collection of management data of UI / UX, individually monitors each configuration, and links between preset settings. Provides information management, overall system processing performance and integrated monitoring information.

Security system 600 performs authentication, authorization, access control for the user and manages the security of the data itself and the security of the transmission path.

The application system 3 processes and provides screens and data necessary for the user based on the smart factory platform 1000.

Those skilled in the art to which the present invention pertains will understand that the above-described present invention can be implemented in other specific forms without changing the technical spirit or essential features.

Therefore, it is to be understood that the embodiments described above are exemplary in all respects and not restrictive. The scope of the present invention is shown by the following claims rather than the detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. do.

1: data acquisition device 2: network
3: Application System 1000: Smart Factory Platform
100: interface system 200: distributed parallel processing system
210: real time processing unit 220: memory unit
230: performance module distribution unit 300: big data analysis system

Claims (24)

In a distributed parallel processing system that processes data for continuous processes in real time for processing data generated in a continuous process in which a plurality of processes are connected,
A plurality of real-time mapping the process identifier of the process in which the collection data is generated for tracking the collected data collected from the continuous process, and sorting the mapped mapping data for linkage processing between the collected data collected in different processes A processing unit;
A plurality of memory units in which the sorted mapping data are distributed and stored; And
And a performing module distribution unit configured to distribute the process identifier mapping and the mapping data in the plurality of real time processing units according to the load of each real time processing unit.
The plurality of real-time processing unit includes a data alignment performing module for performing a unit alignment to align the mapping data by the material unit processed in the process,
The data sorting performing module may determine a collection position at which the collection data is collected using at least one of a length of the material processed in the process, a moving speed of the material, and a collection period of the collection data, and determine the collection position. Distributed parallel processing system for real-time processing the data for the continuous process, characterized in that the mapping data is arranged in a unit. The method of claim 1,
The process identifier mapping is a distributed parallel process for real-time processing of continuous process data, characterized in that at least one mapping between the equipment identifier mapping of each equipment performing the process and the material identifier mapping of the material processed through the equipment. system. The method of claim 1,
The performing module distribution unit,
Distributing the process identifier mapping and the mapping data arrangement to adjust the load of each real-time processing unit by using at least one of an average value of system resources and a usage pattern of system resources according to execution of the execution module distributed to each real-time processing unit. A distributed parallel processing system that processes data for continuous processes in real time. The method of claim 3,
The system resource may include at least one of CPU usage, memory usage, network communication amount, and disk input / output throughput of each real-time processing unit. The method of claim 2,
The plurality of real-time processing unit includes a facility mapping module for mapping the equipment identifier to the collection data and a material mapping module for mapping the material identifier to the data mapped to the equipment identifier; Distributed parallel processing system that processes in real time. The method of claim 5,
The facility mapping performing module processes the continuous process data in real time, wherein the facility mapping function maps the extracted facility identifier to the collected data based on a time when the collected data is collected and attribute information of a sensor that collects the collected data. Distributed parallel processing system. The method of claim 5,
The material mapping performing module is a distributed parallel processing system for real-time processing the continuous process data, characterized in that for mapping the material identifier extracted based on the work instruction information for each process to the collection data. delete The method of claim 1,
The data sorting performing module further performs time sorting to sequentially sort mapping data mapped to the same material identifier among the mapping data according to a collection time. Parallel processing system. delete delete In a distributed parallel processing system that processes data for continuous processes in real time for processing data generated in a continuous process in which a plurality of processes are connected,
A plurality of real-time mapping the process identifier of the process in which the collection data is generated for tracking the collected data collected from the continuous process, and sorting the mapped mapping data for linkage processing between the collected data collected in different processes A processing unit;
A plurality of memory units in which the sorted mapping data are distributed and stored; And
And a performing module distribution unit configured to distribute the process identifier mapping and the mapping data in the plurality of real time processing units according to the load of each real time processing unit.
The plurality of real-time processing unit includes a data alignment performing module for performing a unit alignment to align the mapping data by the material unit processed in the process,
The data sorting performing module
Unit-align the mapping data and the reference data in one direction on the material processed in the process,
The reference data are distributed and parallel processing system for real-time processing the data for the continuous process, characterized in that determined based on the collection point of the collection data and the reference point of the material. In a distributed parallel processing system that processes data for continuous processes in real time for processing data generated in a continuous process in which a plurality of processes are connected,
A plurality of real-time mapping the process identifier of the process in which the collection data is generated for tracking the collected data collected from the continuous process, and sorting the mapped mapping data for linkage processing between the collected data collected in different processes A processing unit;
A plurality of memory units in which the sorted mapping data are distributed and stored; And
And a performing module distribution unit configured to distribute the process identifier mapping and the mapping data in the plurality of real-time processing units according to the load of each real-time processing unit.
The plurality of real-time processing unit includes a data alignment performing module for performing a unit alignment to align the mapping data by the material unit processed in the process,
The data sorting performing module,
At least one of a length of a material processed in the process, a moving speed of the material, and a collection period of the collection data is used to determine a collection position at which the collection data is collected and set reference points set at predetermined intervals on the material. And aligning the mapping data on the basis of the reference data at the respective reference points calculated based on the distance between the respective collection positions. The method of claim 2,
The plurality of real-time processing unit is a distributed parallel processing system for real-time processing the data for the continuous process, characterized in that it comprises a data correction performing module for generating missing collection data, if the collection data is missing. The method of claim 14,
The data correction performing module, when the collection data of a specific region is missing, generates the collection data of the front region or the rear region closest to the region where the omission occurred as the collection data of the space where the omission occurred. Distributed parallel processing system that processes data for continuous processes in real time. The method of claim 14,
The data correction performing module, when the collection data of a specific time point is missing, continuous process data, characterized in that the collection data of the time point immediately before or immediately after the omission occurs as the collection data of the time when the omission occurs Distributed parallel processing system that processes in real time. The method of claim 12,
The time alignment data in which the mapping data is aligned in time, the reference data at the reference points, and the unit alignment data in which the mapping data are sequentially arranged in one direction of the material are separately stored in the plurality of memory units. Distributed parallel processing system for processing data for continuous processing in real time, characterized in that. In a distributed parallel processing system that processes data for continuous processes in real time for processing data generated in a continuous process in which a plurality of processes are connected,
A plurality of real-time mapping the process identifier of the process in which the collection data is generated for tracking the collected data collected from the continuous process, and sorting the mapped mapping data for linkage processing between the collected data collected in different processes A processing unit;
A plurality of memory units in which the sorted mapping data are distributed and stored; And
And a performing module distribution unit configured to distribute the process identifier mapping and the mapping data in the plurality of real-time processing units according to the load of each real-time processing unit.
The plurality of real-time processing unit aligns the mapping data and the reference data in one direction on the material processed in the process,
The reference data are determined based on the collection position of the collection data and the reference point of the material,
One direction on the material is at least one of the longitudinal direction of the material, the width direction of the material, and the thickness direction of the material, distributed parallel processing system for processing data for the continuous process in real time. The method of claim 12,
The first reference data for the material processed in the first step and the second reference data for the material processed in the second step are included in the first material identifier and the second reference data included in the first reference data. A distributed parallel processing system for processing data for continuous processing in real time, characterized in that associated with the second material identifier. The method of claim 2,
The plurality of real-time processing unit includes a facility abnormality detection performing module for determining whether or not abnormality of the facility by comparing a predetermined value of the facility failure determination criteria according to the facility identifier and the measurement value included in the mapping data mapped to the facility identifier; Determination of abnormality of the facility is distributed and parallel processing system for real-time processing of the continuous process data, characterized in that the plurality of real-time processing unit is distributed. The method of claim 2,
The plurality of real-time processing unit includes a quality abnormality detection performing module for determining whether or not the quality abnormality of the material by comparing the quality abnormality determination criteria set in accordance with the material identifier and the mapping data mapped to the material identifier, the quality of the material Determination of abnormality is a distributed parallel processing system for processing data for continuous processing in real time, characterized in that the plurality of real-time processing is distributed in a real time. In a distributed parallel processing system that processes data for continuous processes in real time for processing data generated in a continuous process in which a plurality of processes are connected,
A plurality of real-time mapping the process identifier of the process in which the collection data is generated for tracking the collected data collected from the continuous process, and sorting the mapped mapping data for linkage processing between the collected data collected in different processes A processing unit;
A plurality of memory units in which the sorted mapping data are distributed and stored; And
And a performing module distribution unit configured to distribute the process identifier mapping and the mapping data in the plurality of real time processing units according to the load of each real time processing unit.
The plurality of memory units include a first memory unit and a second memory unit including a master instance and a slave instance,
If the sorted mapping data is stored in the master instance of the first memory unit, the same data is replicated to the slave instance of the second memory unit. If the sorted mapping data is stored in the master instance of the second memory unit, the same data is stored. Is replicated to the slave instance of the first memory unit,
If an error occurs in the master instance of each memory unit, one of the slave instances of the remaining memory unit is a distributed parallel processing system for processing data for the continuous process in real time characterized in that it operates as a master instance. In a distributed parallel processing system that processes data for continuous processes in real time for processing data generated in a continuous process in which a plurality of processes are connected,
A plurality of real-time for mapping the process identifier of the process in which the collection data is generated for tracking the collected data collected from the continuous process, and sorting the mapped mapping data for linkage processing between the collected data collected in different processes A processing unit;
A plurality of memory units in which the sorted mapping data are distributed and stored; And
And a performing module distribution unit configured to distribute the process identifier mapping and the mapping data in the plurality of real time processing units according to the load of each real time processing unit.
The plurality of memory units includes a first memory unit, a second memory unit, and a third memory unit including a master instance, a first slave instance, and a second slave instance.
When the sorted mapping data is stored in the master instance of the first memory unit, the same data is copied to the first slave instance of the second memory unit and the third memory unit, and the mapped data arranged in the master instance of the second memory unit. Is stored, the same data is duplicated in the first slave instance of the first memory unit and the second slave instance of the third memory unit, and the same data is stored in the first instance of the third memory unit. Also replicated to the second slave instance of the memory portion and the second slave instance of the second memory portion,
If an error occurs in the master instance of each memory unit, one of the slave instances of the remaining memory unit is a distributed parallel processing system for processing data for the continuous process in real time characterized in that it operates as a master instance. The method of claim 23, wherein
The master instance, the first slave instance, and the second slave instance are configured in the form of a single thread,
And a plurality of memory units are clustered. The distributed parallel processing system for processing data for a continuous process in real time.

Images