CN114020814A - Process industrial manufacturing full-chain data integration and analysis method and system - Google Patents

Abstract

The application discloses a method and a system for integrating and analyzing full-chain data in process industrial manufacturing, wherein the method comprises the following steps: acquiring data in various business application systems and hardware equipment in the process industry; classifying and summarizing the acquired data according to different theme services, wherein each classified and summarized data corresponds to one theme service; receiving a data analysis requirement; and acquiring the classified summarized data required by the data analysis demand according to the target of the data analysis demand, and analyzing the acquired classified summarized data to obtain a target result. The method and the device solve the problem that the whole chain data of the process industry in the prior art are mutually independent in systems among different enterprises, and therefore multi-source heterogeneous data cross-domain integration and dynamic perception of manufacturing the whole chain in the process industry are achieved.

Description

Process industrial manufacturing full-chain data integration and analysis method and system

Technical Field

The application relates to the field of intelligent manufacturing, in particular to a method and a system for integrating and analyzing full-chain data in process industrial manufacturing.

Background

The process industry is an important industry, and with the development of the internet, process industry enterprises are rapidly developing towards the direction of informatization construction, so that a plurality of information systems with unified standards, such as ERP, MES and DCS, are formed, and the basic application requirements of the enterprises in the aspects of process control, production management, financial management and control and the like are met.

However, systems of different enterprises of a full chain of the same process industry are mutually independent, data formats are not unified, time and space are not matched, models are inconsistent, information is not integrated, and sharing is lacked, so that a large amount of data accumulated by enterprise informatization construction is not fully utilized, deeper data mining and application are lacked, and the systems cannot be directly used for making intelligent decisions of the full chain of enterprises.

Disclosure of Invention

The embodiment of the application provides a method and a system for integrating and analyzing process industry manufacturing full-chain data, which are used for at least solving the problem caused by mutual independence of the process industry full-chain data among different enterprises in the prior art.

According to one aspect of the application, a process industrial manufacturing full-chain data integration and analysis method is provided, which comprises the following steps: acquiring data in various business application systems and hardware equipment in the process industry, wherein each business application system is a system for providing software support for the process industry, and the hardware equipment is production equipment in the process industry; classifying and summarizing the acquired data according to different theme services, wherein each classified and summarized data corresponds to one theme service; receiving a data analysis requirement; and acquiring the classified summarized data required by the data analysis demand according to the target of the data analysis demand, and analyzing the acquired classified summarized data to obtain a target result.

Further, acquiring data in various business application systems and hardware devices in the process industry includes: acquiring real-time data in the process industry; and acquiring historical data in the process industry.

Further, obtaining historical data in the process industry comprises: the historical data is obtained from enterprise relational databases and/or logs in the process industry.

Further, the target outcome includes at least one of the following for a product in the process industry: quality, energy consumption, material consumption, price, supply and demand.

Further, analyzing the obtained classified summarized data according to the target of the data analysis demand to obtain a target result includes: selecting a corresponding machine learning model based on a neural network according to a target result to be analyzed, wherein the machine learning model is obtained by training according to acquired classified and summarized data, and the machine learning model outputs the target result; and calling the machine learning model to obtain the target result according to the data to be analyzed carried in the data analysis requirement.

According to another aspect of the present application, there is also provided a process industry manufacturing full-chain data integration and analysis system, including: the system comprises a data acquisition module, a data processing module and a data processing module, wherein the data acquisition module is used for acquiring data in various business application systems and hardware equipment in the process industry, each business application system is a system for providing software support for the process industry, and the hardware equipment is production equipment in the process industry; the data integration module is used for classifying and summarizing the acquired data according to different theme services, wherein each classified and summarized data corresponds to one theme service; the data analysis module is used for receiving data analysis requirements; and acquiring the classified summarized data required by the data analysis demand according to the target of the data analysis demand, and analyzing the acquired classified summarized data to obtain a target result.

Further, the data acquisition module is configured to: acquiring real-time data in the process industry; and acquiring historical data in the process industry.

Further, the data acquisition module is configured to: the historical data is obtained from enterprise relational databases and/or logs in the process industry.

Further, the target outcome includes at least one of the following for a product in the process industry: quality, energy consumption, material consumption, price, supply and demand.

Further, the data analysis module is configured to: selecting a corresponding machine learning model based on a neural network according to a target result to be analyzed, wherein the machine learning model is obtained by training according to acquired classified and summarized data, and the machine learning model outputs the target result; and calling the machine learning model to obtain the target result according to the data to be analyzed carried in the data analysis requirement.

In the embodiment of the application, data in various business application systems and hardware devices in the process industry are acquired, wherein each business application system is a system for providing software support for the process industry, and the hardware devices are production devices in the process industry; classifying and summarizing the acquired data according to different theme services, wherein each classified and summarized data corresponds to one theme service; receiving a data analysis requirement; and acquiring the classified summarized data required by the data analysis demand according to the target of the data analysis demand, and analyzing the acquired classified summarized data to obtain a target result. The method and the device solve the problem that the whole chain data of the process industry in the prior art are mutually independent in systems among different enterprises, and therefore multi-source heterogeneous data cross-domain integration and dynamic perception of manufacturing the whole chain in the process industry are achieved.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate embodiments of the application and, together with the description, serve to explain the application and are not intended to limit the application. In the drawings:

FIG. 1 is a flow chart of a process industrial manufacturing full chain data integration and analysis method according to an embodiment of the present application;

FIG. 2 is a schematic diagram of a process industrial manufacturing full-chain data integration and analysis system according to an embodiment of the present application.

Detailed Description

It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.

It should be noted that the steps illustrated in the flowcharts of the figures may be performed in a computer system such as a set of computer-executable instructions and that, although a logical order is illustrated in the flowcharts, in some cases, the steps illustrated or described may be performed in an order different than presented herein.

In the present embodiment, a method for integrating and analyzing full-chain data in process industrial manufacturing is provided, and fig. 1 is a flowchart of a method for integrating and analyzing full-chain data in process industrial manufacturing according to an embodiment of the present application, as shown in fig. 1, the flowchart includes the following steps:

step S102, acquiring data in various business application systems and hardware equipment in the process industry, wherein each business application system is a system for providing software support for the process industry, and the hardware equipment is production equipment in the process industry;

in this step, real-time data and/or historical data in the process industry is obtained. Optionally, the acquiring historical data in the process industry includes: the historical data is obtained from enterprise relational databases and/or logs in the process industry.

Step S104, classifying and summarizing the acquired data according to different theme services, wherein each classified and summarized data corresponds to one theme service;

step S106, receiving data analysis requirements;

step S108, obtaining the classified summarized data required by the data analysis demand according to the target of the data analysis demand, and analyzing the obtained classified summarized data to obtain a target result.

In this step, the target result includes at least one of the following of the products in the process industry: quality, energy consumption, material consumption, price, supply and demand.

In this step, a target result may be obtained through machine learning, for example, a corresponding neural network-based machine learning model is selected according to the target result to be analyzed, where the machine learning model is trained according to the obtained classified summary data, and the machine learning model outputs the target result; and calling the machine learning model to obtain the target result according to the data to be analyzed carried in the data analysis requirement.

This is exemplified below. In this example, real-time production data of a production facility is acquired, and a currently desired target result is a change in quality of a product based on the production data. The production data are input into a corresponding machine learning model, the machine learning model outputs the quality change of the product, wherein the machine learning model is obtained by training based on historical data, the production data in the historical data are used as input and output, the quality change corresponding to the production data is used as output data, a pair of input data and the output data are used as a set of training data, and the machine learning model can be obtained by training a plurality of sets of the training data.

The method solves the problem caused by mutual independence of the system of the whole chain data of the process industry among different enterprises in the prior art, thereby realizing multi-source heterogeneous data cross-domain integration and dynamic perception of manufacturing the whole chain in the process industry.

In an alternative embodiment, a process industrial manufacturing full-chain data integration and analysis system is provided, comprising:

and the data acquisition module extracts data from various business application systems and production equipment in the process industry industrial gathering area to realize the real-time updating of the primary summary of the basic data of the data warehouse and the database. Optionally, collecting historical offline data, which is generally extracted from a relational database and a business log of an enterprise; and acquiring real-time production data, and acquiring key process data influencing upper-layer decision.

And the data integration module designs an ODS layer, a DWD layer and a DWS layer in the data warehouse, then designs a DWM layer, and organizes the data into a storage structure facing different theme service dimensions based on OLAP. Optionally, the ODS data access layer basically maintains a format consistent with that of the data source, does not perform excessive verification, and ensures no data loss; and a DWD data detail layer is used for cleaning and converting data, performing dimension compensation and the like, and standardizing.

And the DWS data summarization layer constructs a statistical index with standard naming and consistent caliber according to the upper application theme analysis.

And the DWM data mart layer is used for summarizing related data aiming at a specific application theme.

The data analysis module is used for carrying out feature extraction and correlation analysis on data required by upper-layer decision analysis; model training including data preprocessing, model training, model verification and testing is carried out on the basis of intelligent prediction and intelligent perception of product quality, energy consumption, material consumption, price, supply and demand and the like in the industry.

The data interaction module is used for combining and building a background by using three different types of servers in consideration of safety and expansibility, so that user management, authority management, safety management, log management and function management are realized; for example, the server may include at least one of: logic processing server, static processing server, dynamic forwarding server.

The task scheduling system uses an open source task scheduling tool to complete task scheduling management of real-time and off-line acquisition, integration, analysis and interactive design of a data warehouse, and ensures high-performance operation of the system.

In this embodiment, in order to further perform more optimal management on the system, the system may further include at least one of the following modules:

and the cluster management module is used for providing cluster service for the system background distributed computer cluster.

And the authority management module not only sets system access authority for users of the system application layer, but also for system development and maintenance personnel, so that the system safety is ensured.

And the log management module is used for dividing different events of the system into different grades and storing or prompting the events in log modes of different degrees.

The present embodiment will be described below with reference to the accompanying drawings. Fig. 2 is a schematic structural diagram of a process industrial manufacturing full-chain data integration and analysis system according to an embodiment of the present application, and referring to fig. 2, the system is generally divided into: the system comprises six parts of data source, data acquisition, data integration, data analysis, data interaction and system application.

The data source is as follows: the system data come from upstream and downstream enterprises in the process industry, and the data types comprise real-time updated data in hardware equipment in the production process of upstream and downstream in the industry, and production business data in application systems such as ERP, CRM, EAM, SCM and the like.

Data acquisition: the real-time data of the hardware equipment can be directly read based on the OPC-UA protocol; data in an enterprise relational database and an enterprise business log can be imported into an HDFS file storage system respectively by using Sqoop and flash open-source data extraction tools.

Data integration: a complete system warehouse is constructed after data acquisition, and data integration is divided into four layers: the ODS data access layer performs data synchronization, basically keeps the same format with the source data, and ensures that no data is lost; a DWD layer data detail layer is used for cleaning and converting data, realizing incomplete dimensionality, standardizing, performing exception handling and the like; the DWS data summarization layer provides a theme-oriented and complete history level description for a data source; the DWM data market layer is used for intelligent prediction and intelligent perception in the aspects of product quality, energy consumption, material consumption, price, supply and demand and the like in the industry.

Specifically, the data integration and storage process is based on a Hadoop open-source ecological distributed architecture, and the acquired data are stored in the HDFS; the background computing engine adopts Hive to perform off-line processing, and Flink performs real-time processing; the processed upper layer data is stored in MySQL and cached through Redis and MangoDB for external access.

Optionally, the data integration is based on a Hadoop distributed architecture, and a cluster management module may be developed to provide cluster management for a cluster server formed by a plurality of high-performance computers in the system.

And (3) data analysis: the analysis process is developed based on Python language, an open source machine learning algorithm library is used, and the analysis process comprises the following steps:

and performing dimensionality reduction feature extraction on the data by using methods such as principal component analysis and a self-encoder, and performing correlation analysis based on algorithms such as Aprior and FP-G.

Model training including data preprocessing, model training, model verification and testing is carried out on the basis of intelligent prediction and intelligent perception of product quality, energy consumption, material consumption, price, supply and demand and the like in the industry.

Specifically, when indexes related to time series, such as energy consumption, material consumption, price and the like, are used as input data, a Variation Modal Decomposition (VMD) signal decomposition means is used for decomposing the time series of the input data into characteristic time series, including trend time series (representing the long-term variation trend of the input data), periodic time series (representing the periodic variation trend generated by the input data along with the time variation), fluctuation/noise time series (representing the unpredictable fluctuation generated by the input data due to various emergency events); secondly, learning implicit characteristics in different sequences by using methods such as a Recurrent Neural Network (RNN), a long-short term memory artificial neural network (LSTM), a gated learning unit (GRU), a Transformer model and the like, balancing model training effects and prediction effects by using methods such as an AIC information criterion, a BIC information criterion, an L1 regularization, an L2 regularization, a Dropout method, a Bayesian method and the like, and preventing overfitting of the model; and training the prediction model independently based on each group of feature sequences obtained by decomposition, and weighting the output of the model to obtain the final prediction result of the input data.

When indexes directly influencing enterprise decision making, such as supply and demand, are used as input data, basic influence factors influencing the input data are selected, then deep level features and change modes in the basic influence factors are mined by using methods such as a deep neural network, model training efficiency is improved by using a heuristic algorithm and an AIC information criterion, an intelligent perception model is finally obtained, and changes of input data trends are predicted, so that enterprise operation decision making is guided.

Data interaction: and (3) building a Web server by using Django + Nginx + uWSGI to provide an API (application programming interface) for an application layer, wherein the API comprises user management, authority management, safety management, log management, function management and the like.

For example, Django serves as a logic interface server and provides logic operation of an application function interface; nginx is used as a static processing server and can be used as a reverse proxy to play a role in load balancing; the uWSGI is used as a dynamic forwarding server, the static request of the application layer is directly processed by Nginx, and the dynamic request is forwarded to the uWSGI and then forwarded to Django for processing.

The system application comprises the following steps: the front-end application realizes complex functions through the matching use of different algorithm components, is divided into three blocks of collaborative design, collaborative manufacturing and collaborative service, and covers most application scenes of process industrial enterprises.

And the task scheduling system develops a complete task scheduling system aiming at an acquisition module, an integration module, an analysis module and an interaction module of system data by using an open source scheduling tool AirFlow, and ensures that the system runs automatically and efficiently.

Optionally, on the basis of the above embodiment, the system further includes a rights management module. The system access authority is set for the users of the system application layer and the system development maintainers, and the system safety is ensured.

Optionally, on the basis of the above embodiment, the system further includes a log management module, which classifies different events of the system into different grades and manages the events in the modes of file output, screen output, mailbox reminding, and the like.

Based on the specific implementation, the system adopts a low-coupling and high-cohesion mode as much as possible in the overall design, develops a data cross-domain sharing, manufacturing resource cross-domain management and service flow cross-domain integration and analysis system of the whole process industrial manufacturing chain, and can basically meet the application of most of standard enterprises of the process industrial manufacturing chain.

In this embodiment, an electronic device is provided, comprising a memory in which a computer program is stored and a processor configured to run the computer program to perform the method in the above embodiments.

The programs described above may be run on a processor or may also be stored in memory (or referred to as computer-readable media), which includes both non-transitory and non-transitory, removable and non-removable media, that implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.

These computer programs may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks, and corresponding steps may be implemented by different modules.

Such an apparatus or system is provided in this embodiment. The system is called a process industry manufacturing full-chain data integration and analysis system, and comprises: the system comprises a data acquisition module, a data processing module and a data processing module, wherein the data acquisition module is used for acquiring data in various business application systems and hardware equipment in the process industry, each business application system is a system for providing software support for the process industry, and the hardware equipment is production equipment in the process industry; the data integration module is used for classifying and summarizing the acquired data according to different theme services, wherein each classified and summarized data corresponds to one theme service; the data analysis module is used for receiving data analysis requirements; and acquiring the classified summarized data required by the data analysis demand according to the target of the data analysis demand, and analyzing the acquired classified summarized data to obtain a target result.

The system or the apparatus is used for implementing the functions of the method in the foregoing embodiments, and each module in the system or the apparatus corresponds to each step in the method, which has been described in the method and is not described herein again.

For example, the data acquisition module is configured to: acquiring real-time data in the process industry; and acquiring historical data in the process industry. Optionally, the data acquisition module is configured to: the historical data is obtained from enterprise relational databases and/or logs in the process industry.

As another example, the target outcome includes at least one of the following for a product in the process industry: quality, energy consumption, material consumption, price, supply and demand. Optionally, the data analysis module is configured to: selecting a corresponding machine learning model based on a neural network according to a target result to be analyzed, wherein the machine learning model is obtained by training according to acquired classified and summarized data, and the machine learning model outputs the target result; and calling the machine learning model to obtain the target result according to the data to be analyzed carried in the data analysis requirement.

Through above-mentioned embodiment design a process industrial manufacturing full chain integration and analytic system, rely on open source distributing type big data storage and calculation tool to build data warehouse, include: the data acquisition module is used for extracting data from various business application systems and production equipment in the process industry gathering area to be uniformly stored; the data integration module is used for designing an ODS layer, a DWD layer, a DWS layer and a DWM layer in a data warehouse; the data analysis module is used for carrying out feature extraction and correlation analysis on the data, carrying out model training aiming at problems such as intelligent prediction, intelligent perception and the like in industrial intelligence and providing decisions for the upper layer of an enterprise; the data interaction module uses three different types of Web servers to provide more efficient application service for an upper layer in consideration of safety and expansibility; in addition, a complete task scheduling system is designed for real-time and off-line acquisition, integration, analysis and interaction of the data warehouse, and high-performance operation of the system is guaranteed.

The above are merely examples of the present application and are not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims (10)

1. A process industrial manufacturing full-chain data integration and analysis method is characterized by comprising the following steps:

acquiring data in various business application systems and hardware equipment in the process industry, wherein each business application system is a system for providing software support for the process industry, and the hardware equipment is production equipment in the process industry;

classifying and summarizing the acquired data according to different theme services, wherein each classified and summarized data corresponds to one theme service;

receiving a data analysis requirement;

and acquiring the classified summarized data required by the data analysis demand according to the target of the data analysis demand, and analyzing the acquired classified summarized data to obtain a target result.

2. The method of claim 1, wherein obtaining data in various business application systems and hardware devices in the process industry comprises:

acquiring real-time data in the process industry;

and acquiring historical data in the process industry.

3. The method of claim 2, wherein obtaining historical data in the process industry comprises:

the historical data is obtained from enterprise relational databases and/or logs in the process industry.

4. The method of any one of claims 1 to 3, wherein the target outcome comprises at least one of the following for a product in the process industry: quality, energy consumption, material consumption, price, supply and demand.

5. The method according to claim 4, wherein analyzing the obtained classified summary data according to the target of the data analysis requirement to obtain a target result comprises:

selecting a corresponding machine learning model based on a neural network according to a target result to be analyzed, wherein the machine learning model is obtained by training according to acquired classified and summarized data, and the machine learning model outputs the target result;

and calling the machine learning model to obtain the target result according to the data to be analyzed carried in the data analysis requirement.

6. A process industry manufacturing full-chain data integration and analysis system, comprising:

the system comprises a data acquisition module, a data processing module and a data processing module, wherein the data acquisition module is used for acquiring data in various business application systems and hardware equipment in the process industry, each business application system is a system for providing software support for the process industry, and the hardware equipment is production equipment in the process industry;

the data integration module is used for classifying and summarizing the acquired data according to different theme services, wherein each classified and summarized data corresponds to one theme service;

the data analysis module is used for receiving data analysis requirements; and acquiring the classified summarized data required by the data analysis demand according to the target of the data analysis demand, and analyzing the acquired classified summarized data to obtain a target result.

7. The system of claim 6, wherein the data acquisition module is configured to:

acquiring real-time data in the process industry;

and acquiring historical data in the process industry.

8. The system of claim 7, wherein the data acquisition module is configured to:

the historical data is obtained from enterprise relational databases and/or logs in the process industry.

9. The system of any one of claims 6 to 8, wherein the target outcome comprises at least one of the following for a product in the process industry: quality, energy consumption, material consumption, price, supply and demand.

10. The system of claim 9, wherein the data analysis module is configured to:

selecting a corresponding machine learning model based on a neural network according to a target result to be analyzed, wherein the machine learning model is obtained by training according to acquired classified and summarized data, and the machine learning model outputs the target result;

and calling the machine learning model to obtain the target result according to the data to be analyzed carried in the data analysis requirement.

Images