KR102210972B1 - Smart Factory System for Performing High Speed Big Data Analysis - Google Patents

Abstract

A smart factory system for performing high-speed big data analysis according to an aspect of the present invention capable of providing a big data analysis service in real time includes: a middleware system for preprocessing collected data collected from a data collection device; A distributed parallel processing system for generating mapping data by mapping a process identifier of a process in which the collected data is generated to the pre-processed collected data, and generating alignment data by aligning the generated mapping data according to a predetermined criterion; And a high-speed big data analysis system for analyzing the alignment data using a big data analysis model, wherein the high-speed big data analysis system includes: a first big data storage unit in which the alignment data is stored in an instance in a binary format; A second big data storage unit for storing the alignment data stored in the first big data storage unit as a file in the disk; And analyzing the alignment data stored in the first big data storage unit using a real-time big data analysis model, and analyzing the alignment data stored in the second big data storage unit using a non-real time big data analysis model. It characterized in that it comprises a device.

Description

Smart Factory System for Performing High Speed Big Data Analysis {Smart Factory System for Performing High Speed Big Data Analysis}

The present invention relates to a factory management system, and more specifically, to a smart factory.

Each process is related to each other, such as a plurality of processes for producing a finished product using raw materials are continuously performed, and the outputs of each process are mixed with each other or the state of the output of a specific process is changed and supplied to the subsequent process. The production method is called a continuous process production method. The steel industry, energy industry, paper industry, or oil refinery industry are representative industries to which the continuous process production method is applied.

In the case of industries to which such a continuous process production method is applied, unlike industries to which a single process production method is applied, raw materials or intermediate materials move at high speed, so the data collection cycle is short and the amount of data is large, as well as noise, dust and moisture. Since products are produced in a factory environment with many lights, measurement errors often occur, and intermediate materials are mixed with each other or the location of materials is moved depending on the work method.

Accordingly, in the case of an industry to which a continuous process production method is applied, a system capable of processing a lot of data in real time and processing data generated by each process is required.

However, a general factory data processing system disclosed in Korean Patent Application Publication No. 10-2015-0033847 (name of invention: digital factory capacity management system reflecting real-time factory conditions, published on April 2, 2015), etc. Since the processing system) is for processing and analyzing data generated in a single process, it cannot process a lot of data generated in a continuous process in real time, and it is also possible to analyze the relationship between the data generated by each process. There is no problem.

In particular, in the case of the existing factory data processing system, while the generated data is being recorded, the data cannot be accessed until the recording operation is completed, and the data cannot be loaded even after the data recording operation is completed. Since it takes a lot of time, there is a problem in that it is not possible to perform an analysis task requiring real-time analysis such as quality determination or abnormality prediction of materials or products.

The present invention is to solve the above-described problem, and it is an object of the present invention to provide a smart factory system that performs high-speed big data analysis capable of providing a big data analysis service in real time.

In addition, the present invention is another technical problem to provide a smart factory system that performs high-speed big data analysis capable of connecting a requester to a high-speed big data analysis system using an API.

A smart factory system for performing high-speed big data analysis according to an aspect of the present invention for achieving the above object includes: a middleware system for preprocessing collected data collected from a data collection device; A distributed parallel processing system for generating mapping data by mapping a process identifier of a process in which the collected data is generated to the pre-processed collected data, and generating alignment data by aligning the generated mapping data according to a predetermined criterion; And a high-speed big data analysis system for analyzing the alignment data using a big data analysis model, wherein the high-speed big data analysis system includes: a first big data storage unit in which the alignment data is stored in an instance in a binary format; A second big data storage unit for storing the alignment data stored in the first big data storage unit as a file in the disk; A file generation unit generating the alignment data stored in the first big data storage unit as the file when a predetermined event occurs and storing the alignment data in the second big data storage unit; And analyzing the alignment data stored in the first big data storage unit using a real-time big data analysis model, and analyzing the alignment data stored in the second big data storage unit using a non-real time big data analysis model. It characterized in that it comprises a device.

According to the present invention, the big data storage unit is divided into a first big data storage unit that provides data in real time for big data analysis and a second big data storage unit that can provide long-term large-capacity data, It has the effect of being able to provide analysis services.

In addition, according to the present invention, it is possible to access the high-speed big data analysis system using the API, so even if there is a change in the infrastructure or solution of the high-speed big data analysis system, there is no need to change the software connecting the requester and the high-speed big data analysis system. It has the effect of reducing development cost.

1 is a diagram showing a smart factor architecture according to an embodiment of the present invention.
2 is a block diagram showing the configuration of a middleware system according to an embodiment of the present invention.
3 is a diagram showing the configuration of a middleware system including a plurality of processing units and a plurality of queue storage units.
4 is a diagram showing in detail the configuration of a distributed parallel processing system according to an embodiment of the present invention.
5 is a diagram showing the configuration of a distributed parallel processing system including a plurality of processing units and a plurality of memory units.
6 is a conceptual diagram illustrating a method of distributed parallel processing of data mapping and sorting tasks by way of example.
7 is a diagram showing a configuration of a tripled memory unit.
8 is a diagram showing in detail the configuration of a high-speed big data analysis system according to an embodiment of the present invention.
9 is a diagram showing an example of a configuration of a first or second big data storage unit shown in FIG. 8.
10 is a diagram showing in detail the configuration of an API service providing system according to an embodiment of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

The meaning of the terms described in this specification should be understood as follows. Singular expressions should be understood as including plural expressions unless clearly defined differently in context, and terms such as “first” and “second” are used to distinguish one element from other elements, The scope of rights should not be limited by these terms.

It is to be understood that terms such as "comprise" or "have" do not preclude the presence or addition of one or more other features or numbers, steps, actions, components, parts, or combinations thereof.

The term “at least one” is to be understood as including all possible combinations from one or more related items. For example, the meaning of “at least one of the first item, the second item, and the third item” means 2 among the first item, the second item, and the third item, as well as the first item, the second item, and the third item. It means a combination of all items that can be presented from more than one.

1 is a diagram showing a smart factory architecture according to an embodiment of the present invention.

As shown in FIG. 1, the smart factory architecture 10 according to the present invention includes a data collection system 1, a network system 2, and a smart factory platform 1000.

The data collection system 1 collects data generated in the process. In one embodiment, the data collection system 1 may collect data generated in a continuous process in which a plurality of processes are successively performed. The continuous process refers to a process in which a plurality of processes for producing a finished product using raw materials are continuously performed and the outputs of each process are mixed with each other or the state of the outputs of a specific process is changed and supplied to a subsequent process. The steel process is a representative example of such a continuous process. Hereinafter, for convenience of explanation, it is assumed that the data collection system 1 is applied to a continuous process such as a steel process.

Since the steel process consists of various processes such as the iron making process, the steel making process, the casting process, and the rolling process, the data collection system 1 is in the process of various processes such as the iron making process, steel making process, casting process, and rolling process. Collect the generated micro data (Micro Data). Here, the micro data refers to raw data as data collected through various sensors. In the following, for convenience of explanation, micro data collected in a continuous process will be indicated as collected data.

The data collection system 1 includes various measuring instruments, sensors, actuators, etc. for collecting data generated in a continuous process. The data collection system 1 may further include a P/C, a programmable logic controller (PLC), a distributed control system (DCS), or the like that integrates or controls data collected by a measuring instrument, a sensor, an actuator, or the like.

The network system 2 delivers the collected data to the smart factory platform 1000. The network system 2 may include a network cable, a gateway, a router, or a wireless access point (AP).

The smart factory platform 1000 receives collected data through the network system 2. The smart factory platform 1000 processes the collected data, determines whether there is an abnormality in equipment or materials based on the processed collected data, and provides a search and analysis service for stored data.

In one embodiment, the smart factory platform 1000 according to the present invention includes a middleware system 100, a distributed parallel processing system 200, a high-speed big data analysis system 300, and an API as shown in FIG. 1. It includes a service providing system 400.

The middleware system 100 preprocesses the collected data. The middleware system 100 is connected to Level 0 to Level 2 devices. Hereinafter, the middleware system 100 according to the present invention will be described in more detail with reference to FIGS. 2 and 3.

2 is a block diagram showing a configuration of a middleware system according to an embodiment of the present invention, and FIG. 3 is a diagram illustrating a configuration of a middleware system including a plurality of processing units and a plurality of queue storage units.

In more detail with reference to FIGS. 2 and 3, the middleware system 100 includes an interfacing unit 110 and a queue server 120. In addition, the middleware system 100 may further include a middleware management unit 130 and a queue management unit 140,

The interfacing unit 110 pre-processes the collected data for cooperative processing of the collected data. The interfacing unit 110 may pre-process the collected data by standardizing the collected data. To this end, the interfacing unit 110 includes at least one of the parsing unit 112, the standardization unit 113, the filtering unit 114, and the transmission unit 115.

The parsing unit 112 parses the collected data and generates parsed data. The collected data may have a structure in which a group ID including a plurality of item IDs, a collection time, and a plurality of measurement values are repeated. In this case, the item ID is for identifying the measured attribute, and is a value indicating which of the attributes of equipment, materials, or products has been measured, and may be temperature or humidity. The group ID may be a representative value of a group of several items for each location or each process in a specific factory. The collection time may be included in the group ID.

When the collected data is received in a form in which a group ID, a collection time, and a plurality of measured values are repeatedly received without a separate distinction, the parsing unit 112 stores the collected data based on a preset layout for cooperative processing of the collected data. Parse.

The parsing unit 112 parses the collected data by group ID, and matches a plurality of item IDs included in the group ID with a plurality of measured values, respectively, and parses data in a form having a single item ID, collection time, and a single measurement value. Can be created.

The parsing unit 112 may parse the collected data based on a message layout for the entire collection data.

The standardization unit 113 standardizes the parsed data to generate standardized data. For each parsing data, the standardization unit 113 converts the item ID included in each parsed data into a standard item ID according to a preset standard conversion criterion, and unifies the unit and number of digits of the measurement value included in the parsed data. By doing so, you can standardize the parsed data. In this case, the preset standard conversion criterion may include a standard item ID set for each item ID such as various sensors, and a reference unit and number of digits according to each standard item ID.

The standardization unit 113 may change the item ID included in each parsed data to the standard item ID so that data having the same measured attribute have the same item ID.

The standardization unit 113 pre-processes the parsed data so that the parsed data having the same measured attribute among the parsed data have the same standard item ID, so that the data collected in a continuous process can be linked and processed based on the standard item ID.

The filtering unit 114 selects standardized data to be stored in the queue server 120 from among standardized data according to a preset filtering criterion. For example, a grade is preset according to the type of standardized data, and the filtering unit 114 may select standardized data to be stored in the queue server 120 according to the grade. In an embodiment, the grade may be determined according to importance based on the standard item ID of standardized data. The filtering unit 114 transmits the selected standardized data to the transmission unit 115.

The transmission unit 115 stores standardized data provided from the filtering unit 114 in the queue server 120. The transmission unit 115 may store standardized data in the queue storage unit 121 of the queue server 120 for each group ID or standard item ID.

The transmission unit 115 may store standardized data in the queue storage unit 121 having a low load in consideration of the loads of the plurality of queue storage units 121. In another embodiment, if the queue server 120 in which standardized data is stored for each factory or process among the plurality of queue servers 120 is preset, the transmission unit 115 transmits the standardized data to the corresponding standardized data. It is stored in the set queue server 120.

The transmission unit 115 may determine whether to store standardized data according to the operation mode of the interfacing unit 110. Specifically, the transmission unit 115 stores standardized data in the queue server 120 periodically when the operation mode of the interfacing unit 110 is in the normal mode, and standardizes when the operation mode of the interfacing unit 110 is in the standby mode. You can stop saving data. In this case, the operation mode of the interfacing unit 110 may be determined based on the number of queue servers 120 operating normally among the plurality of queue servers 120.

The interfacing unit 110 may further include a data merging unit 116. The data merging unit 116 merges and transmits the collected data to the parsing unit 112 to improve data processing performance. In one embodiment, the data merging unit 116 merges collected data received at a predetermined time interval (eg, 0.1 second, 1 second, 1 minute, etc.).

Due to the nature of the continuous process, the collected data can be transferred to the parsing unit 112 in a very short period (e.g., 5 ms to 20 ms). The data merging unit 116 does not merge the collected data necessary for monitoring and It is transferred directly to 112, and the remaining collected data may be merged at regular time intervals and transferred to the parsing unit 112.

In this case, whether the data is collected data necessary for monitoring may be set according to the importance of the collected data. For example, when an abnormality occurs, collected data collected from facilities or materials that require immediate action may be set as collected data necessary for monitoring.

In one embodiment, the middleware system 100 may further include a middleware management unit 130 and a queue management unit 140 to manage the operation mode of the interfacing unit 110.

The middleware management unit 130 determines whether the queue server 120 is normally operating in the operation check unit 131, and determines the operation mode of the interfacing unit 110 in the mode management unit 132.

Through this, the availability of the interfacing unit 110 can be increased, and a secondary failure of the interfacing unit 110 can be prevented by actively responding to a failure of the queue server 120.

The operation check unit 131 determines whether the plurality of queue servers 120 operate normally. In an embodiment, the operation check unit 131 may determine whether the plurality of queue servers 120 operate normally based on a response to the test signal. The operation check unit 131 determines that the corresponding queue server 120 does not operate normally when there is no response from the queue server 120 or a response in a predetermined format is not received from the queue server 120 to the test signal.

The mode management unit 132 determines an operation mode of the interfacing unit 110 based on the operation states of the plurality of queue servers 120. The mode management unit 132 transmits the operation mode of the interfacing unit 110 to the interfacing unit 110.

In one embodiment, the mode management unit 132 may determine an operation mode of the interfacing unit 110 based on the number of the queue servers 120 operating normally among the plurality of queue servers 120. Specifically, the mode management unit 132 compares the amount of collected data received from the data collection system 1 with the number of queue servers 120 operating normally, and the amount of collected data received is a queue server operating normally ( When the capacity of 120) is not exceeded, the operation mode of the interfacing unit 110 may be determined as a normal mode. In this case, the amount of collected data may mean an average value of the amount of collected data received in real time or the amount of collected data periodically received.

For example, the mode management unit 132 may determine an operation mode of the interfacing unit 110 as shown in Table 1 below.

Number of queue storage Number of queue storage units that operate normally Number of queue storage units that operate abnormally Operation mode 3 3 0 Normal mode 3 2 One Caution mode 3 One 2 Standby mode 3 0 3 Standby mode

When the mode management unit 132 determines the normal mode, the interfacing unit 110 stores standardized data in the queue server 120 set in advance among the plurality of queue servers 120. When the mode management unit 132 determines the attention mode, the interfacing unit 110 stores standardized data in the queue servers 120 other than the queue server 120 operating abnormally. When the mode management unit 132 determines the standby mode, the interfacing unit 110 stops receiving collected data and storing standardized data. At this time, in the caution mode, some of the plurality of queue servers 120 do not operate normally. It represents an operation mode in a situation in which the interfacing unit 110 can store standardized data in real time through the remaining queue server 120. The mode management unit 132 may determine the operation mode as the attention mode when at least two or more queue servers 120 among the plurality of queue servers 120 operate normally.

The queue management unit 140 is provided to correspond to each of the plurality of queue servers 120. The queue management unit 140 manages meta data for each queue server 120 and checks whether the queue server 120 operates normally. To this end, the queue management unit 140 may include a meta data management unit 141.

The meta data management unit 141 manages meta data for the queue storage unit 121 of the corresponding queue server 120. As an example, the meta data management unit 141 manages meta data such as basic specification information, access information, and configuration information such as topics and partitions, and the topics and partitions to store data to the interfacing unit 110 based on the meta data. You can provide information about

When some of the plurality of queue servers 120 do not operate normally, the interfacing unit 110 can quickly find the location of the storable queue server 120 using the metadata of the queue server 120 and store the parsed data. have.

The queue server 120 temporarily stores standardized data before processing it in real time. To this end, the queue server 120 includes at least one queue storage unit 121.

The queue storage unit 121 is a storage unit for storing standardized data for a predetermined period of time, and may store data based on a disk rather than a memory to prevent data loss. The space for storing data in the queue storage unit 121 may be divided into topics, and the queue storage unit 121 may separate several partitions within the same topic and process them in parallel.

In one embodiment, as shown in FIG. 3, a plurality of queue servers 120 may be implemented, and a plurality of queue servers 120 may be clustered with each other. In this case, when the transmission unit 115 stores the parsed data in any one of the plurality of queue servers 120, the same parsed data is also stored in the other queue servers 120.

In one embodiment, the parsed data stored in the queue server 120 may be assigned a unique group ID for each data group fetched by the distributed parallel processing system 200 from the queue server 120. Accordingly, data fetch addresses can be managed for each unique group ID, so that data can be stored and provided in the form of a queue that reads and writes data sequentially.

According to this embodiment, the plurality of interfacing units 110 may be expanded in a form of addition depending on the size of the data collection system 1 and the physical location of the factory, and each interfacing unit 110 is highly available ( It can be implemented in a redundant structure for high availability (HA).

In addition, when standardization of the collected data is completed, the interfacing unit 110 selects one queue server 120 from among the plurality of queue servers 120 and stores the standardized data. At this time, the criterion for selecting the queue server 120 to store the standardized data may be selected from a variety of rules, for example, selecting the queue server 120 having the lowest load, a method of sequentially selecting, or collecting data. It is possible to pre-store and select the queue server 120 to be stored for each collected sensor.

In addition, as shown in FIG. 3, each interfacing unit 110 may each include a middleware management unit 130. Each middleware management unit 130 determines whether or not the plurality of queue storage units 120 operate normally, and determines an operation mode of the corresponding interfacing unit 110.

Referring back to FIG. 1, the distributed parallel processing system 200 processes standardized data transmitted from the middleware system 100. In one embodiment, the distributed parallel processing system 200 generates mapping data in which process identifiers are mapped to standardized data, and uses the mapping data in each process as a predetermined standard for linkage processing of collected data collected for each process. Sort according to.

Hereinafter, the configuration of the distributed parallel processing system 200 according to the present invention will be described in more detail with reference to FIGS. 4 to 7.

4 is a diagram showing in detail the configuration of a distributed parallel processing system according to an embodiment of the present invention. As shown in FIG. 4, the distributed and parallel processing system 200 according to an embodiment of the present invention includes a processing unit 210 and a memory unit 220.

The processing unit 210 generates mapping data by mapping process identifiers to standardized data, and arranges the mapping data so that data between areas such as operation-facility-quality can be linked and analyzed. In addition, the processing unit 210 predicts a point where no data has been collected because there is no sensor or missing data that occurs in the middle of the collection period.

To this end, the processing unit 210 includes at least one of a fetching unit 211, a process mapping unit 213, a data correction unit 215, and a data alignment unit 216. do. In addition, the processing unit 210 may further include a facility abnormality detection performing unit 217 and a quality abnormality detection performing unit 218.

In one embodiment, a plurality of execution units 211 to 218 shown in FIG. 4 may be distributed to the processing unit 210 to be implemented as an application that implements each function, and these applications are After creating a work space in 210, a plurality of threads are created to perform functions assigned to each execution unit 211 to 218.

The fetch execution unit 211 reads standardized data from the queue storage unit 121 of the middleware system 100 and stores it in the memory unit 220. The fetch execution unit 211 may read the next data of the previously read data by storing location information of previously inquired data for each of the plurality of queue storage units 121.

At this time, when the interfacing unit 110 stores the standardized data in the queue storage unit 121 for each group ID or standard item ID for linking processing of collected data collected in a continuous process, the fetch execution unit 211 stores the queue. Standardized data stored in the unit 121 is read for each group ID or standard item ID.

The process mapping execution unit 213 generates mapping data by mapping a process identifier for identifying a process in which the corresponding standardized data is collected to the standardized data read by the fetch performing unit 211.

In one embodiment, the process mapping execution unit 213 generates first mapping data by mapping a facility identifier of a facility performing each process to standardized data, or standardizes the material identifier of a material processed by the facility. Alternatively, second mapping data may be generated by mapping to the first mapping data. To this end, the mapping unit 213 may include a facility mapping unit 213a and a material mapping unit 213b.

The facility mapping execution unit 213a generates first mapping data by mapping the facility identifier of the facility for which the corresponding standardized data is collected to the standardized data. The facility mapping execution unit 213a may acquire a facility identifier to be mapped to the standardized data based on a collection time at which the standardized data is collected or attribute information of a sensor that has collected the standardized data. In one embodiment, the facility identifier may be a facility number assigned to each facility.

The material mapping unit 213b stores the material identifier of the material processed in the facility for which the standard data is collected in the standardized data read from the memory unit 220 or the first mapping data generated by the facility mapping unit 213a. Mapping to generate second mapping data. The material mapping execution unit 213b acquires the material identifier of the material generated through the facility for which the corresponding standardized data is collected based on the work instruction information performed in each process, and maps the obtained material identifier to the first mapping data. .

In one embodiment, the material identifier may be a material number assigned to each material.

The data correction performing unit 215 corrects the mapping data by adding missing data among the mapping data. The data correction performing unit 215 may correct the missing data by using the mapping data at the location closest to the area where the missing data should have been collected, or the mapping data at the collection time closest to the time when the missing data should have been collected. .

In an embodiment, the data correction performing unit 215 may match a collection time included in the mapping data with a predetermined collection period to correct the mapping data. For example, when continuous process data is stored with a collection cycle of 20 ms, the data correction unit 215 corrects the collection time of the mapping data with a collection time of 15:01:11 seconds 0005 ms to 15:01:11 seconds 0000 ms, and , The collection time of the mapping data with a collection time of 15:01:11 seconds 0050ms can be corrected to 15:01:11 seconds 0040ms.

The data alignment unit 216 arranges the mapping data or the corrected mapping data for linking processing between data of each process.

The data alignment performing unit 216 generates first alignment data by arranging mapping data to which the same material identifier is mapped in a chronological order in a material unit for linking processing between collected data collected in a continuous process.

The data alignment performing unit 216 sorts the first alignment data on a material corresponding to the same material identifier based on a collection location at which the data is collected to generate second alignment data.

In this case, the collection position may be determined using at least one of the length of the material, the moving speed of the material, and the collection period of the collected data. For example, the data sorting unit 216 may determine a collection position at which collection data is collected for each period on the material based on a value obtained by multiplying the movement speed of the material by the collection period and the total length of the material. Accordingly, the data alignment unit 216 may align the first alignment data into data measured at a predetermined position in one direction on the material.

The data alignment performing unit 216 is used for linking processing of the collected data collected from the first and second processes at different collection cycles, between reference points set at predetermined intervals on each material and the collection location of the second alignment data. A measurement value at each reference point is calculated based on the distance, and reference data at each reference point is generated based on the calculated measurement value.

The data alignment performing unit 216 sequentially arranges the second alignment data and the reference data at the reference points on the material in one direction. In one embodiment, one direction may be at least one of a length direction of a material, a width direction of a material, and a thickness direction of a material.

Hereinafter, an example in which the data alignment performing unit 216 arranges reference data on a material in the longitudinal direction will be described in detail.

First reference points are set at predetermined intervals in the longitudinal direction of the first material processed in the first process, and second reference points are set at predetermined intervals in the longitudinal direction of the second material processed in the second process. In this case, the first material identifier corresponding to the first material is mapped to the first reference data at the first reference points, and the second material identifier corresponding to the second material is mapped to the second reference data at the second reference points. The identifier is mapped. Accordingly, the first reference data and the second reference data are linked based on the first material identifier and the second material identifier on a material family tree (not shown) in which a material identifier is mapped for each material.

In other words, each material identifier is connected in a tree form in the material family tree, and mapping data of each process is displayed through the material identifier assigned to the material that is generated while sequentially passing through the first process and the second process by referring to this material family tree. They can be linked to each other.

The data alignment unit 216 stores the second alignment data and reference data arranged in the length direction of the material in the memory unit 220 as described above.

In this way, the processing unit 210 maps a process identifier such as a facility identifier or a material identifier to standardized data, and arranges the mapping data so that the collected data collected in a continuous process can be linked and processed.

The facility abnormality detection execution unit 217 receives the first mapping data from the equipment mapping execution unit 213a, and determines whether or not there is a facility abnormality according to a predetermined equipment abnormality determination criterion. As a result of the determination, if it is determined that an abnormality has occurred in a specific facility, the facility abnormality detection unit 217 stores the determination result in the memory unit 220.

The quality abnormality detection performing unit 218 determines whether or not there is a quality abnormality according to a predetermined quality abnormality determination criterion based on the second alignment data arranged by the data alignment execution unit 216. As a result of the determination, when it is determined that an abnormality has occurred in the quality of a specific material, the quality abnormality detection unit 218 stores the determination result in the memory unit 220.

In one embodiment, the quality abnormality detection performing unit 218 generates macro data to be used as a reference for a quality abnormality determination formula through operations such as average and error prediction of the second alignment data, and 2 By entering the alignment data into the quality abnormality determination formula, quality abnormality can be determined according to the result.

In the above-described embodiment, it has been described that the distributed parallel processing system 200 maps and arranges standardized data through one processing unit 210 and one memory unit 220, but in a modified embodiment, the distributed parallel processing system 200 The processing system 200 may map and arrange standardized data using a plurality of processing units 210a, 210b, and 210c and a plurality of memory units 220 as shown in FIG. 5.

Hereinafter, a distributed parallel processing system according to a modified embodiment will be described with reference to FIGS. 4 and 5.

5 is a diagram schematically showing the configuration of a distributed parallel processing system including a plurality of processing units and a plurality of memory units. The distributed parallel processing system 200 includes a plurality of processing units 210a, 210b, 210c, a plurality of memory units 220a, 220b, 220c, and an execution unit distribution unit 230.

One or more execution units 211 to 218 among the plurality of execution units 211 to 218 for mapping and sorting standardized data are distributed and distributed to the plurality of processing units 210a, 210b, and 210c. The plurality of processing units 210a, 210b, and 210c include a fetching unit 211, a facility mapping unit 213a, a material mapping unit 213b, a data correction unit 215, and a data alignment unit 216. , By distributing and parallel processing at least one of the equipment abnormality detection execution unit 217, and the quality abnormality detection execution unit 218, and storing the final result data in the memory unit 220, the middleware system 100 Standardized data transmitted from) is processed in real time.

In an embodiment, the plurality of processing units 210a, 210b, and 210c may be configured in a clustering structure. As described above, since the plurality of processing units 210a, 210b, and 210c have a clustering structure, when a failure occurs in a specific processing unit, the execution units 211 to 218 running in the failed processing unit can be moved to another processing unit. Availability can be ensured.

Data processed by the plurality of processing units 210a, 210b, and 210c are stored in the plurality of memory units 220. In an embodiment, in order to increase processing performance and ensure availability in case of failure, the plurality of memory units 220 may have a clustering structure like the above-described queue storage unit 121.

The plurality of memory units 220 may be provided in a redundant structure for high availability (HA). That is, each memory unit 220 includes a master instance (M) and a slave instance (S). In this case, the master instance M included in the first memory unit 220a and the slave instance S included in the second memory unit 220b operate as a pair, and are included in the second memory unit 220b. The master instance M and the slave instance S included in the first memory unit 220a may operate as a pair.

As for the alignment data recorded in the slave instance S, each data may be backed up as a scripter-type file for recovery in case of a failure. In this case, the scripter type file means a file in which commands related to writing or reading data are stored together with the data.

The master instance M and the slave instance S of each memory unit 220 are configured in a single thread (Rhread) type, and instances and ports may be separated for each write and read.

Hereinafter, a method of performing distributed parallel processing of standardized data mapping and alignment will be described with reference to FIG. 6 as an example.

As shown in FIG. 6, since the fetch execution unit 211 is distributed in the first processing unit 210a, the first processing unit 210a executes the fetch execution unit 211 to perform the fetch execution unit 211. ) Connects to the queue storage unit 121 to fetch standardized data, and stores the fetch data in the master instance M of the first memory unit 220a. At this time, data is also copied and stored in the slave instance S of the second memory unit 220b. Since the equipment mapping execution unit 213a and the material mapping execution unit 213b are distributed to the second processing unit 210b, the second processing unit 210b executes the equipment mapping execution unit 213a to perform a second memory. The equipment identifier is mapped to the standardized data read from the slave instance S of the unit 220b or the slave instance S of the first memory unit 220a, and the material mapping execution unit 213b is executed to perform standardized data or equipment. The material identifier is mapped to the first mapping data to which the identifier is mapped.

Since the data correction unit 215 and the data alignment unit 216 are distributed in the third processing unit 210c, the third processing unit 210c executes the data correction unit 215 The missing data is corrected, and the data alignment performing unit 216 is performed to align the mapping data or the corrected mapping data in units of material and store them in the master instance M of the second memory unit 220b. At this time, data is also stored in the slave instance S of the first memory unit 220a.

In the above-described embodiment, the plurality of memory units 220 are configured to be duplicated, so that the master instance M included in the first memory unit 220a and the slave instance S included in the second memory unit 220b Is operated as a pair, and the master instance M included in the second memory unit 220b and the slave instance S included in the first memory unit 220a operate as a pair.

However, according to this embodiment, since the master instance (M) and the slave instance (S) are implemented as a single thread, when the master instance (M) of the first memory unit 220a is down, the first memory unit ( During the downtime until the master instance M of 220) is normalized, the slave instance S of the second memory unit 220b cannot service both the write operation and the read operation.

Accordingly, in a modified embodiment, the memory unit 220 may be implemented in a triplex structure, as shown in FIG. 7. Specifically, each of the memory units 220 according to the modified embodiment includes a master instance M, a first slave instance S1, and a second slave instance S2.

The master instance M included in the first memory unit 220a operates as a pair with the first slave instances S1 of the second and third memory units 220b and 220c. Accordingly, if data is written to the master instance M included in the first memory unit 220a, the data is also copied and stored in the first slave instances S1 of the second and third memory units 220b and 220c. do.

In addition, the master instance M included in the second memory unit 220b is a first slave instance S1 included in the first memory unit 220a and a second slave instance included in the third memory unit 220c. It works in pairs with (S2). Accordingly, when data is written to the master instance M included in the second memory unit 220b, it is included in the first slave instance S1 and the third memory unit 220c included in the first memory unit 220a. Data is also stored in the second slave instances S2.

Also, the master instance M included in the third memory unit 220c operates as a pair with the second slave instances S1 included in the first memory unit 220a and the second memory unit 220b. . Accordingly, when data is written to the master instance M included in the third memory unit 220c, the second slave instances S1 included in the first memory unit 220a and the second memory unit 220b are also The data is copied and stored.

Referring back to FIG. 5, the execution unit distribution unit 230 distributes a plurality of execution units 211 to 218 to a plurality of processing units 210a to 210c. In addition, the execution unit distribution unit 230 may perform a plurality of execution units 211 according to the load of the processing units 210a to 210c due to execution of the execution units 211 to 218 distributed to the processing units 210a to 210c. ~218) may be redistributed in the processing units 210a to 210c.

Specifically, the performing unit distribution unit 230 includes a performing unit storage unit 232, a distribution order determining unit 234, and a distribution performing unit 236.

First, the execution unit storage unit 232 stores a plurality of execution units 211 to 218 for performing mapping and alignment of standardized data.

Distribution order determination unit 234, after the plurality of execution units (211 to 218) are distributed to each processing unit (210a to 210c) by the distribution performing unit 236, the resource usage information of the processing units (210a to 210c) Is determined, a distribution order for redistribution of the plurality of execution units 211 to 218 is determined so that the load amount of the plurality of processing units 210a to 210c can be adjusted. In an embodiment, the distribution order determination unit 234 may determine a distribution order for redistribution of the plurality of execution units 211 to 218 so that the loads of the plurality of processing units 210a to 210c are equalized. Here, the determination of the distribution order means determining the processing units 210a to 210c to which the respective execution units 211 to 218 are to be distributed.

In another embodiment, the distribution order determination unit 234 distributes a plurality of execution units 211 to 218 using at least one of an average value of system resources and a usage pattern of system resources according to execution of the first deployed execution unit. You can decide the order. According to this embodiment, the system resource may include at least one of a CPU usage rate, a memory usage, a network communication amount, and a disk input/output throughput of each of the processing units 210a to 210c.

The distribution performing unit 236 distributes and distributes the plurality of execution units 211 to 218 to the plurality of processing units 210a to 210c according to the distribution order determined by the distribution order determination unit 234.

Specifically, the distribution execution unit 236 randomly distributes the plurality of execution units 211 to 218 stored in the execution unit storage unit 232 to the processing units 210a to 210c. Thereafter, when a predetermined pause is reached, the distribution performing unit 236 retrieves the execution units 211 to 218 distributed to each of the processing units 210a to 210c and stores them in the execution unit storage unit 232, and when the pause is completed, The plurality of execution units 211 to 218 stored in the execution unit storage unit 232 are distributed to the corresponding processing units 210a to 210c according to the distribution order determined by the distribution order determination unit 234.

Referring back to FIG. 1, the high-speed big data analysis system 300 stores the alignment data arranged by the distributed parallel processing system 200 in a big data storage space. In particular, the high-speed big data analysis system 300 according to the present invention separates and stores the sorted data in two big data storage units in the big data storage space so that a big data analysis service can be provided in real time.

Hereinafter, a high-speed big data analysis system 300 according to the present invention will be described in detail with reference to FIG. 8.

8 is a block diagram showing the configuration of a high-speed big data analysis system according to an embodiment of the present invention. As shown in FIG. 8, the high-speed big data analysis system 300 according to an embodiment of the present invention includes a data processing unit 310, a big data storage unit 320, a query processing unit 330, and an analysis device 340. ).

The data processing unit 310 is a distributed parallel processing of the alignment data and the abnormality detection result, and a completion event receiving unit 311, an alignment data fetch unit 312, a memory queue 313, a data storage unit 314a, and a file generation At least one of the unit 314b, the abnormality detection data receiving unit 315, and the metadata generating unit 316 is included, and the big data storage unit 320 stores alignment data by the data processing unit 310 As a space, a first big data storage unit 322, a second big data storage unit 324, and a model storage unit 326 are included.

The completion event receiving unit 311 monitors the memory unit 220 of the distributed parallel processing system 200 and transmits the completion event to the alignment data fetch unit 312 when a completion event is newly stored.

When the completion event is transmitted from the completion event receiving unit 311, the alignment data fetch unit 312 inquires the alignment data corresponding to the completion event in the memory unit 220 and stores it in the memory queue 313. In one embodiment, the alignment data fetch unit 312 uses the key information included in the completion event to check in which partition and directory of the memory unit 220 data corresponding to the completion event is stored. Data stored in the unit 220 may be inquired and stored in the memory queue 313.

The memory queue 313 temporarily stores the alignment data read by the alignment data fetch unit 312 in the memory before storing it in the big data storage unit 320.

The data storage unit 314a stores the alignment data stored in the memory queue 313 in a binary form in an instance of the first big data storage unit 322. In an embodiment, the data storage unit 314a may change or modify the alignment data stored in the memory queue 313 and store it in the first big data storage unit 322. In addition, the data storage unit 314a may change or modify the alignment data stored in the first big data storage unit 322.

The file generation unit 314b generates the alignment data stored in the first big data storage unit 322 as a physical file and stores it in a disk of the second big data storage unit 324. In one embodiment, when a predetermined event occurs, the file generation unit 314b may generate the alignment data stored in the first big data storage unit 322 as a file and store it in the second big data storage unit 324. .

For example, the file generation unit 314b may generate the alignment data that has passed a predetermined period among the alignment data stored in the first big data storage unit 322 as a file and store it in the second big data storage unit 324. . As another example, the file generation unit 314b generates, as a file, the alignment data that has been modified or changed and becomes cold data among the alignment data stored in the first big data storage unit 322 as a file. It can be stored in the storage unit 324.

Meanwhile, when the file generation unit 314b stores the generated file in the second big data storage unit 324, it may compress the size of the file or change the format of the file and store it.

In one embodiment, the file generating unit 314b may be implemented as a plurality of file generating units. Through this, the plurality of file generation units 314a to 314n can process the file generation operation in parallel, thereby improving the speed of the file generation operation. According to this embodiment, the plurality of file generation units 314b may be clustered with each other.

In the above-described embodiment, when storing the alignment data as a physical file, the file generation unit 314b may divide and store the file in a plurality of block types. Accordingly, the file generation unit 314b may generate metadata for specifying the location where each block is stored, and manage files stored in the second big data storage 324 by using the generated metadata. In this case, to generate metadata, the file generation unit 314b may create and allocate an index for each block, and may specify each block using the index within the metadata.

In the above-described embodiment, the data storage unit 314a stores all sorting data stored in the memory queue 313 in the first big data storage unit 322, and the file generation unit 314b stores the first big data. It has been described that the alignment data stored in the unit 322 is recorded in the second big data storage unit 324 in the form of a file.

However, in a modified embodiment, the data storage unit 314a has a reference value in which the number of columns or rows is less than the reference value when writing to the table among the alignment data stored in the memory queue 313. Sorting data that is less than or equal to is selected and stored in the first big data storage unit 322, and sorted data with a number of columns or rows greater than or equal to a standard value or sort data whose size is greater than or equal to a standard value is directly generated by the file generation unit 314b in a file It may be stored in the second big data storage unit 324.

In the above-described embodiment, among the sorting data stored in the memory queue 313 by the data storage unit 314a, only sorted data in which the number of columns or rows is less than the reference value or the size is less than the reference value are stored in the first big data storage unit 322 The reason for this is that if the number of columns or rows is greater than or equal to the reference value or if the sorted data whose size is greater than or equal to the reference value is recorded in the form of a table in the first big data storage unit 322, it may take a lot of time to load, thus providing real-time big data service Because this can be difficult.

The abnormality detection data receiving unit 315 monitors the memory unit 220 of the distributed parallel processing system 200 and stores a new abnormality detection result in the memory queue 313 when a new abnormality detection result is stored.

The metadata generation unit 316 maps information on the identifier of the table included in the first big data storage 322, the position on the table, and columns and rows in which each sorted data is stored in the corresponding table for each sorted data. Create and save data.

In this case, the file generation unit 314b may determine data to be generated as a file among data recorded in each table based on the metadata generated by the metadata generation unit 316.

In addition, the metadata generation unit 316 additionally records the location of the file generated by the file generation unit 314b, the file name, the block ID in which the file is stored, and the storage location of the server in the metadata. For example, when a file is created in the file generation unit 314b, the metadata generation unit 316 records the location and file name of the file in the metadata, and the file is larger than the block size and divides into 5 blocks. When stored in, 15 block IDs and storage locations of each server are additionally stored in metadata.

In the first big data storage unit 322, at least some of the alignment data stored in the memory queue 313 are stored in the instance in a binary form by the data generation unit 314a. That is, at least some of the alignment data stored in the memory queue 313 through the data generation unit 314a is recorded in a binary form in the instance included in the first big data storage unit 322.

As described above, the reason that the high-speed big data analysis system 300 according to the present invention stores the alignment data in the first big data storage unit 322 in binary form is that when the alignment data is stored as a file, the alignment data is viewed in real time. Because this is impossible. Specifically, when the sort data is saved in the form of a file, the sort data is saved in units of time or day, but the file cannot be accessed until the file saving operation is closed. Real-time confirmation is impossible.

Moreover, when storing the sorted data as a file, when the file is divided into blocks and saved as the size of the file is large, the file storage location must be managed as separate metadata. To do this, indexes must be allocated for each block as well as data browsing. For this, it is necessary to load separate meta data and index, and because it takes a lot of time to load meta data and index, real-time confirmation of data becomes impossible.

Accordingly, the high-speed big data analysis system 300 according to the present invention may enable real-time confirmation of the alignment data by primarily storing the alignment data in an instance of the first big data storage unit 322 in a binary format. .

The alignment data selected by the file generation unit 314b among the alignment data stored in the first big data storage unit 322 are stored in the second big data storage unit 324 in a physical file format.

In the above-described embodiment, the big data storage unit 320 including the first and second big data storage units 322 and 324 may be implemented based on a distributed file system.

Meanwhile, the first and second big data storage units 322 and 324 may be configured of a master node 320a and a data node 320b, respectively, as shown in FIG. 9. The master node 320a stores a table in which the sorting data is recorded or a file of sorting data in the data nodes 320b, and creates and manages a job for querying the table or file stored in the data node 320b. do.

Here, a job refers to a unit for processing a query received from the query processing device 330 to inquire a table or file stored in the data node 320b.

In particular, the master mode (320a), when executing a job (Job) for querying of data stored in the data node (320b), distribution for each job (Job) and when loading data of a specific file, metadata generation unit (316) The metadata generated by) is used as the location information of the data.

A table or file of sorting data is stored in the data node 320b. A plurality of data nodes 320b may be implemented. In this case, all of the tables or files are stored in a storage unit included in each data node 320b.

Referring back to FIG. 8, at least one big data analysis model for big data analysis is stored in the model storage unit 326. In an embodiment, the big data analysis model may include a real-time big data analysis model and a non-real-time big data analysis model. For example, the real-time big data analysis model is a quality determination model necessary to determine the quality of a material or product, a facility monitoring analysis model for monitoring the facility where the collected data is generated, and whether or not an abnormality occurs in the process where the collected data is generated. It may include at least one of the abnormal occurrence detection model for detecting, and the non-real-time big data analysis model may include at least one of an artificial intelligence learning model and a trend analysis.

In one embodiment, the analysis model may be composed of a statistical formula or may be composed of an AI model.

The query processing unit 330 retrieves the data stored in the big data storage unit 320 and returns the data. To this end, the query processing unit 330 includes a query receiving unit 332, a query execution unit 336, and a query result. And at least one of the transmission units 338. The query processing unit 330 may further include a query scheduling unit 334.

Specifically, the query receiving unit 332 receives a query from a user and interprets the received query syntax.

The query execution unit 336 transmits the query received through the query reception unit 322 to the big data storage unit 320 to execute the query, and obtains a query execution result from the big data storage unit 320. In one embodiment, the query execution unit 336 uses the metadata generated by the metadata generation unit 316 when executing the query, and the corresponding query is the first big data storage unit 322 of the big data storage unit 320. ) Or a query to be transmitted to the second big data storage unit 324 may be determined.

The query result transmission unit 338 transmits the query execution result obtained from the query execution result big data storage unit 320 to the user who requested the query.

On the other hand, when the query received through the query receiving unit 332 is composed of a plurality of subqueries, the query scheduling unit 334 classifies the received query into each subquery and delivers it to the query execution unit 336 do.

In an embodiment, the query receiving unit 332 may receive a query through an application programming interface (API) provided through the API service providing system 400. Specifically, when a request for execution of an API to access the high-speed big data analysis system 300 and a specific event are received from a requester, the API service providing system 400 generates a query corresponding to the event by executing the API. Then, the generated query is transmitted to the query receiving unit 332. In addition, the API service providing system 400 may obtain an execution result corresponding to a corresponding query through an API and provide it to the requester.

When an execution of any one of the at least one big data analysis model stored in the model storage unit 326 is requested from the requester, the analysis device 340 executes the requested big data analysis model, and the corresponding big data Big data analysis results obtained through execution of the analysis model are returned as a request.

To this end, the analysis device 340 includes an analysis model call unit 342, an analysis model execution unit 344, and an analysis result transmission unit 346, as shown in FIG. 8.

When the analysis model execution request is received from the request, the analysis model call unit 342 specifies the model identifier (ID) of the big data analysis model requested to be executed, and an analysis corresponding to the model identifier of the specified big data analysis model. The big data analysis model is called by loading the model from the model storage unit 326.

The analysis model execution unit 344 executes the called big data analysis model by using at least one of the alignment data stored in the first big data storage unit 322 and the file stored in the second big data storage unit 3340. Acquire analysis results.

For example, when a real-time big data analysis model such as a quality determination model, a facility monitoring analysis model, and an abnormality detection model is called through the analysis model calling unit 342, the alignment data stored in the first big data storage unit 322 The big data analysis can be executed in real time by executing the corresponding big data analysis model using.

As another example, when a non-real-time big data analysis model such as an artificial intelligence learning model and trend analysis is called through the analysis model calling unit 342, the analysis model execution unit 344 is stored in the second big data storage unit 324. By executing the corresponding big data analysis model using the saved file, non-real-time big data analysis can be performed. In this case, in the case of trend analysis, the analysis model execution unit 344 may additionally use the alignment data stored in the first big data storage unit 322 to improve accuracy.

The analysis result transmission unit 346 transmits the big data analysis result obtained by the analysis model execution unit 344 as a request.

In an embodiment, the requester may access the analysis device 340 using an API provided through the API service providing system 400. Specifically, when a request for execution of an API that can access the analysis device 340 and a request for execution of a specific analysis model are requested from the requester, the API service providing system 400 executes the corresponding API so that the analysis model call unit 342 Allows you to call a specific analysis model. In addition, the API service providing system 400 may obtain the big data analysis result from the analysis result transmission unit 346 through the API and provide it as a requester.

Referring back to FIG. 1, the API service providing system 400 executes an API for connection between the requester 500 and the respective systems 100, 200, and 300 constituting the smart factory platform 1000. .

Specifically, when an event including identification information and a command of an API corresponding to a target system to be accessed from the requester 500 is received, the API service providing system 400 executes the API to execute the execution result for the event. Is returned to the requester 500.

In the present invention, the reason for providing an API for connection between the requester 500 and each system 100, 200, 300 through the API service providing system 400 is, by providing a standard data interface through the API, each system This is because even if the infrastructure and solutions of (100, 200, 300) are changed, there is no need to develop a separate software accordingly, and software development productivity can be improved.

Specifically, when the requester 500 directly connects to each system 100, 200, 300 through a dedicated interface of each system 100, 200, 300, the server and storage of each system 100, 200, 300 , Or when various infrastructure resources such as a network or an applied open source or library are versioned, a change in software constituting the request 500 is inevitably required.

However, as in the present invention, when the requester 500 accesses each system 100, 200, 300 through an API, even if the infrastructure and solution of each system 100, 200, 300 are changed, the requester 500 Since a change in software for each system is not required, it is possible to minimize the impact caused by changes in infrastructure and solutions of each system 100, 200, 300.

In addition, API services can be provided for each data storage and utilization type, so that efficiency for infrastructure resources can be provided, and faster services can be provided to users.

In addition, in order to obtain data from each system (100, 200, 300), the requester 500 does not directly access each system (100, 200, 300), and is provided for each system (100, 200, 300). Since it is possible to access the corresponding systems 100, 200, and 300 through an API to receive a call service and a feedback service for an event, system access convenience may be improved from the viewpoint of the requester 500.

Hereinafter, the configuration of the API service providing system 400 according to the present invention will be described in more detail with reference to FIG. 10.

10 is a block diagram showing the configuration of an API service providing system according to an embodiment of the present invention. As shown in FIG. 10, the API service providing system 400 according to an embodiment of the present invention includes an API authentication unit 411, an API gateway 412, and an API server 413. In addition, the API service providing system 400 according to an embodiment of the present invention further includes at least one of an API management unit 414, a policy management unit 415, an API logging unit 416, and a log database 417. I can.

First, when an API use request is received from the requester 500, the API authentication unit 411 authenticates whether the requester has the right to use the API. In one embodiment, the API use request may include identification information of the requester 500 and identification information of an API intended to be used by the requester 500.

Specifically, the API authentication unit 411 refers to the API list in which the identification information of the API that can be used by each requester 500 is recorded, and whether the corresponding request 500 can use the API included in the API use request. Check. If it is determined that the request 500 can use the API as a result of the verification, the requester 500 authenticates as a legitimate request and issues a token to the requester 500. At this time, the token serves as an electronic signature that authenticates that the request 500 is authorized to use the API for a certain period of time.

In an embodiment, the API authentication unit 411 may map and store the validity period of the corresponding token and the identification information of the requester 500 that can use the token on a token issuing list when issuing a token.

On the other hand, when the API authentication unit 411 receives a confirmation request for whether a specific token is a valid token from the API gateway 412, the requested token is a valid token by checking the validity period of the requested token on the token issuing list. Check whether it is. As a result of the verification, if the validity of the token is within the validity period recorded on the token issuing list, the API authentication unit 411 generates a verification result that the token is valid and transmits it to the API gateway 412. .

When the API authentication unit 411 receives a confirmation request for whether a specific token is a valid token from the API gateway 412, the requester 500 that can use the token as well as the validity period of the requested token on the token issuing list is received. ), you can also check whether the token is a valid token.

The API gateway 412 receives an API execution request from the request 500. The API execution request includes a token issued by the requester 500 from the API authentication unit 411, identification information of the API to be used by the requester 500, and an event (or command) to be executed using the API. . In this case, the event or command may include a data inquiry request or a query execution request to be performed for each system 100, 200, 300 using an API.

When an API execution request is received from the requester 500, the API gateway 412 requests the API authentication unit 411 to confirm whether the token included in the API execution request is a valid token. When a result of confirming that the token is valid is received from the API authentication unit 411, the API gateway 412 transmits the identification information and events of the API included in the API execution request to the API server 413 so that the API server 413 Calls the API so that the API can be executed based on the event.

In addition, when the execution result according to the execution of the API is received from the API server 413, the API gateway 412 transmits the received execution result to the requester 500.

When the API server 413 receives the API identification information and the event from the API gateway 412, the API server 413 calls and executes the API corresponding to the identification information of the API, so that the event received through the API gateway 412 is transmitted through the API. So that it can be done.

For example, if the API corresponding to the API identification information is an API for accessing the middleware system 100, the API server 413 provides a first API for accessing the middleware system 100 as shown in FIG. It is executed to access the queue storage unit 121 of the middleware system 100, and when the received event is an inquiry request for standardized data recorded in the queue storage unit 121, the queue storage unit 121 through the corresponding API Execution results are obtained by inquiring standardized data.

As another example, when the API corresponding to the API identification information is an API for accessing the memory unit 220 of the distributed parallel processing system 200, the API server 413 is a distributed parallel processing system 200 as shown in FIG. Execute a second API for accessing the memory unit 220 of) to access the memory unit 220 of the distributed parallel processing system 200, and query the alignment data recorded in the memory unit 220 with the received event In the case of a request, an execution result is obtained by accessing the memory unit 220 through the second API and querying the alignment data from the memory unit 220.

In addition, when the API corresponding to the API identification information is an API for accessing the fetching unit 211 of the distributed parallel processing system 200, the API server 413 is a distributed parallel processing system ( By executing a third API for accessing the fetching unit 211 of 200), it is connected to the fetching unit 211 of the distributed parallel processing system 200, and the received event is sent to the distributed parallel processing system 200. In the case of a data transmission request to the fetching unit 211, the data is transmitted to the fetching unit 211 by accessing the fetching unit 211 through a third API.

On the other hand, when the API corresponding to the API identification information is an API for accessing the facility abnormality detection execution unit 217 or the quality abnormality detection execution unit 218 of the distributed parallel processing system 200, the API server 413 As shown in Figure 4, by executing the fourth API for access to the facility abnormality detection execution unit 217 or the quality abnormality detection execution unit 218 of the distributed parallel processing system 200, the distributed parallel processing system 200 Equipment abnormality detection execution unit 217 or quality abnormality detection execution unit 218 connected to the equipment abnormality detection execution unit 217 or quality abnormality detection execution unit 218, and the received event is In case of receiving an event from the facility abnormality detection execution unit 217 or the quality abnormality detection execution unit 218 through the fourth API, the equipment abnormality detection execution unit 217 or the quality abnormality detection execution unit 218 Receive an event.

As another example, when the API corresponding to the API identification information is an API for accessing the query processing unit 330 of the big data analysis system 300, the API server 413 is the analysis system 300 as shown in FIG. When the fifth API for access to the query processing unit 330 of is accessed to access the query processing unit 330 of the analysis system 300, and the received event is a data inquiry request recorded in the big data storage 320, the corresponding Through the execution of the API, a query for a data inquiry request is generated and transmitted to the query processing unit 330, and data obtained by executing the query is received through the API.

On the other hand, when the API corresponding to the API identification information is an API for accessing the analysis device 340 of the big data analysis system 300, the API server 413 analyzes the analysis system 300 as shown in FIG. When the sixth API for access to the device 340 is executed to access the analysis device 340 of the analysis system 300, and the received event is a request for execution of a specific analysis model, the API server 413 Through the execution of, the analysis device 340 can call a specific analysis model, and the big data analysis result according to the execution of the corresponding analysis model is received from the analysis device 340.

The API server 413 provides the execution result obtained by executing the API according to the received event to the requester 500 through the API gateway 412. In an embodiment, the API server 413 may directly transmit the execution result to the requester 500 without passing through the API gateway 412. Specifically, the API server 413 may directly transmit the execution result to the requester 500 when the amount of data in the execution result exceeds a predetermined reference value.

The API management unit 414 adds, deletes, or modifies an API. Specifically, the API management unit 414 may add an API by newly defining a function for the API. In addition, the API management unit 414 may modify an existing API function or delete an API. API functions include information on the system (100, 200, 300) to which each corresponding API will access, information on the requester 500 using the API, query syntax or function to be called, condition value, data to be transmitted, event to be transmitted , It can be configured with parameters including whether to reply, reply type, etc.

The policy management unit 415 generates an API list in which API identification information that can be used by the requester 500 is mapped for each requester 500. In the API list, identification information of each requester 500 and API identification information that can be used by the requester 500 are mapped and recorded.

The API logging unit 416 records execution results and events transmitted and received through the API gateway 412 or the API server 413 in the log database 417. At this time, the API logging unit 416 maps the execution result or the transmission/reception time of the event to the execution result or event and records it in the log database 417.

Referring back to FIG. 1, the requester 500 includes various types of systems or users that transmit events to the smart factory platform 1000 or receive data from the smart factory platform 1000.

In one embodiment, the requester 500 may access the smart factory platform 1000 through an API provided by the API service providing system 400.

The requester 500 may include an app, an application, a job, or a user.

An app refers to software that is installed on the mobile and is in charge of specialized functions or services. An application refers to a legacy system such as MES, SCM, and ERP, and refers to a system that searches or transmits data and events through a web screen, or runs in the form of an AP server without a web screen.

Task refers to a task in a process created to call an API service, and one task can call another task.

Users refer to developers and operators who directly execute queries or call APIs through various analysis tools or BI tools, or users who call platform API services such as developers and operators who call APIs for program development.

Those of ordinary skill in the art to which the present invention pertains will be able to understand that the above-described present invention can be implemented in other specific forms without changing the technical spirit or essential features thereof.

Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

Claims (17)

A middleware system that pre-processes the collected data collected from the data collection device;
A distributed parallel processing system for generating mapping data by mapping a process identifier of a process in which the collected data is generated to the pre-processed collected data, and generating alignment data by aligning the generated mapping data according to a predetermined criterion; And
Including a high-speed big data analysis system for analyzing the alignment data using a big data analysis model,
The high-speed big data analysis system,
A first big data storage unit in which the alignment data is stored in an instance in a binary format;
A second big data storage unit for storing the alignment data stored in the first big data storage unit as a file in the disk;
A file generation unit generating the alignment data stored in the first big data storage unit as the file when a predetermined event occurs and storing the alignment data in the second big data storage unit; And
An analysis device that analyzes the alignment data stored in the first big data storage unit using a real-time big data analysis model, and analyzes the alignment data stored in the second big data storage unit using a non-real-time big data analysis model Smart factory system for performing high-speed big data analysis comprising a. The method of claim 1,
The real-time big data analysis model includes a quality determination model for quality determination of materials or products, a facility monitoring analysis model for monitoring facilities in which the collected data is generated, and whether or not an abnormality occurs in the process in which the collected data is generated. Smart factory system for performing high-speed big data analysis, characterized in that it comprises at least one of the abnormal occurrence detection models for detection. The method of claim 1,
The non-real-time big data analysis model is a smart factory system for performing high-speed big data analysis, characterized in that it includes at least one of an artificial intelligence learning model and a trend analysis. delete The method of claim 1,
The predetermined event is a smart factory system for performing high-speed big data analysis, characterized in that the alignment data is determined after a predetermined period has elapsed or a change of the alignment data is completed. The method of claim 1,
The high-speed big data analysis system
An alignment data fetch unit reading the alignment data from a memory unit of the distributed parallel processing system;
A smart factory system for performing high-speed big data analysis, further comprising a data storage unit for recording the read alignment data in the instance of the first big data storage unit; The method of claim 6,
The file generation unit performs high-speed big data analysis, characterized in that the alignment data exceeding a predetermined size among the alignment data stored in the memory unit is directly generated as the file and stored in the second big data storage unit. Smart factory system. The method of claim 1,
The high-speed big data analysis system further includes a model storage unit storing at least one big data analysis model,
When an execution request for any one of the at least one big data analysis model is received in response to a request from a requester, the analysis device executes the requested big data analysis model to obtain a big data analysis result, Return the obtained big data analysis result to the requester,
An execution request for any one of the big data analysis models is received through an application programming interface (API) for connection between the requester and the high-speed big data analysis system, and the big data analysis result is received through the API. A smart factory system that performs high-speed big data analysis, characterized in that it is provided as a tester. The method of claim 8,
The analysis device,
An analysis model calling unit for calling a corresponding big data analysis model by loading the requested big data analysis model from the model storage unit;
An analysis model execution unit that executes the called big data analysis model to obtain the big data analysis result; And
A smart factory system for performing high-speed big data analysis, further comprising: an analysis result transmission unit for transmitting the analysis result of the big data obtained by the analysis model execution unit to the requester. The method of claim 9,
In the analysis model execution unit, the big data analysis model requested to be executed is a quality determination model for quality determination of materials or products, a facility monitoring analysis model for monitoring the facility in which the collected data is generated, and the collected data are generated. In the case of a real-time big data analysis model including at least one of an abnormality detection model for detecting whether an abnormality occurs in a process, the big data is executed by executing the corresponding analysis model using the alignment data stored in the first big data storage A smart factory system that performs high-speed big data analysis, characterized in that acquiring analysis results. The method of claim 9,
The analysis model execution unit uses a file stored in the second big data storage unit when the requested big data analysis model is a non-real-time big data analysis model for at least one of an artificial intelligence learning model and a trend analysis. Smart factory system for performing high-speed big data analysis, characterized in that to obtain the big data analysis result by executing. The method of claim 1,
The high-speed big data analysis system,
A completion event receiving unit for monitoring a memory unit of the distributed parallel processing system to receive a completion event indicating completion of storing the alignment data; And,
When the completion event is transmitted from the completion event receiving unit, a smart factory for performing high-speed big data analysis, further comprising an alignment data fetching unit for inquiring the memory unit to obtain the alignment data corresponding to the completion event. system. The method of claim 12,
The alignment data fetch unit acquires partition information and directory information of a memory unit in which alignment data corresponding to a corresponding completion event is stored using key information included in the completion event, and refers to the obtained partition information and directory information to the memory. A smart factory system that performs high-speed big data analysis, characterized in that the unit acquires the alignment data. The method of claim 1,
The high-speed big data analysis system further comprises a query processing unit that executes a query input from an external source and returns an execution result for the query. The method of claim 14,
The query is generated through an API for connection between the requester and the high-speed big data analysis system, and the execution result of the query is provided to the requester through the API. Smart factory system. The method of claim 1,
The process identifier mapping includes at least one of a facility identifier mapping of a facility in which the collected data is generated and a material identifier mapping of a material processed through the facility. A smart factory system for performing high-speed big data analysis. The method of claim 1,
The alignment data is generated by arranging mapping data to which the process identifier is mapped according to at least one of a collection time sequence and a material unit.

Images