CN114595773A

CN114595773A - Multi-source heterogeneous twin data fusion method and system based on factory production process

Info

Publication number: CN114595773A
Application number: CN202210237309.7A
Authority: CN
Inventors: 张锐; 郑敬莎; 郭洪飞; 何泰霖; 王燕; 丁新文; 赵锦钰; 杨思妍
Original assignee: Guangdong Taiyunze Technology Co ltd; Jinan University; Tianjin University of Science and Technology
Current assignee: Guangdong Taiyunze Technology Co ltd; Jinan University; Tianjin University of Science and Technology
Priority date: 2022-03-10
Filing date: 2022-03-10
Publication date: 2022-06-07
Anticipated expiration: 2042-03-10
Also published as: CN114595773B

Abstract

A multi-source heterogeneous twin data fusion method based on a factory production process is disclosed, which comprises the following steps: modeling the collected multiple sensor data into a network topological graph of a sensor data time sequence accumulation process based on data classification and semantic analysis of a hierarchical feature aggregation model; according to the characteristics of a time sequence dynamic topological graph structure, a GCN-LSTM structure is adopted to combine structural information with time information, nodes are embedded into a low-dimensional space by utilizing network embedding, meanwhile, the structure and the property of the network are captured, and a hierarchical feature aggregation method is adopted to learn different feature aggregators in neighborhoods of different depths; aggregating information of different depths using LSTM to ensure that information flows from higher depths to nodes; based on the establishment of the incomplete multi-mode deep semantic matching fusion model, a cross-mode deep semantic matching mechanism is adopted, and the incomplete multi-mode deep semantic matching fusion model is established through the multi-layer nonlinear correlation among modal data.

Description

Multi-source heterogeneous twin data fusion method and system based on factory production process

Technical Field

The invention relates to the technical field of intelligent manufacturing, in particular to a multi-source heterogeneous twin data fusion method and system based on a factory production process.

Background

The intelligent manufacturing prompts large-scale production and manufacturing enterprises to transform towards the intelligent direction, wherein, the mass data owned by the factory brings great pressure to the intelligent development of the factory on one hand, and on the other hand, the data are the core driving force of the factory intelligence on the other hand.

The coupling degree of digital twins and intelligent manufacturing is higher and higher, the ground of a digital twins technology is one of key elements for promoting intelligent manufacturing development, heterogeneous data exists in a digital twins system, the data comprises physical entity data generated by physical equipment of a workshop, virtual model data generated by a virtual model corresponding to the physical equipment of the workshop and related to simulation prediction, physical data describing the physical attributes of the physical equipment of the workshop, behavior rule data for describing the process flow and behavior rules of the physical equipment and the like, and how to effectively fuse the heterogeneous data is worthy of research, so that unified service is output to upper-layer users.

The inventor finds that, at present, for the problems existing in twin data fusion, the fusion of entity data and virtual data is realized mainly through a data twin method, and simulation and application are performed in workshops and other intelligent manufacturing fields. However, the current research results do not fundamentally solve the problems of validity and accuracy of data from a factory. The data has the characteristics of high mass, isomerism, multi-source, time-varying and dimensionality, and the fusion effectiveness and accuracy of heterogeneous data are poor due to unbalanced data distribution, incomplete modes, missing or non-normative semantic expression and the like.

Disclosure of Invention

The invention aims to provide a multi-source heterogeneous twin data fusion method and system based on a factory production process, which can effectively fuse heterogeneous twin data of a plurality of sensors in different production processes of a factory so as to output a unified service requirement for upper-layer users, and can improve the effectiveness and accuracy of fusion between multi-source heterogeneous twin data fusion in the factory production process to a certain extent.

In order to achieve the above object, an embodiment of the present invention provides a multi-source heterogeneous twin data fusion method based on a factory production process, including:

s1, collecting sensor data of a plurality of sensors in different production processes of a factory;

s2, data classification and semantic analysis based on the hierarchical feature aggregation model, including:

s21, aiming at the sensor data, modeling the sensor data as a network topological graph of a sensor data time sequence accumulation process; defining edges between sensor nodes as sequential events, and establishing a time sequence dynamic topological graph structure formed by dynamic processes driven by interactive events between the nodes and neighbors of the nodes;

s22, according to the characteristics of the time sequence dynamic topological graph structure, adopting a GCN-LSTM structure to combine structural information with time information, embedding nodes into a low-dimensional space by utilizing network embedding, simultaneously capturing the structure and the property of the network, and adopting a hierarchical feature aggregation method to learn different feature aggregators in neighborhoods of different depths; and aggregating information of different depths using the LSTM to ensure that the information flows from the higher depth to the node;

s3, establishing an incomplete multi-mode depth semantic matching fusion model based on the incomplete multi-mode depth semantic matching fusion model, and establishing the incomplete multi-mode depth semantic matching fusion model by adopting a cross-mode depth semantic matching mechanism and through multi-layer nonlinear correlation among modal data; the method comprises the following steps: constructing a shared characteristic subspace among the modes to learn the sharing of incomplete multi-mode data; local similarity of data of each mode in a shared subspace is ensured by setting a regularization factor of the invariant graph; and establishing a new objective function to describe the incomplete multi-modal data deep semantic matching model.

Preferably, the step S21 further constructs a heterogeneous multi-source sensor network data topology map by:

the set G (V, E, X, L) is a data topological graph of the heterogeneous multi-source sensor network, wherein V is a set of sensor nodes, E is the combination of edges between the nodes,

is a set of node features, while L represents a set of different node labels; the distance between a node v and a node adjacent to v is k, which can be expressed as

It is known that this node embedding can be expressed as

Each embedded depth-k-node embedding can be expressed as

Preferably, the step S22 specifically includes:

s222, embedding the learning nodes through a hierarchical aggregation framework:

first, using an aggregation method similar to GraphSAGE, a set of features for a depth k neighborhood is synthesized into a single vector, as shown in equation (1):

where s (x) gives the sample of nodes in neighborhood set x, AGGREGATE is a feature aggregation similar to GraphSAGE, and K ∈ {1, …, K }, where K is the maximum depth that can be found by the sensor data network; learned node embedding

Is the information of the channel captured from the neighborhood node v' with the distance depth of k;

then, in combination with LSTM metacells, for each node v, it was learned at different depths

And the last node is embedded in h_vIf v is assigned, then equation (2) holds:

finally, the learned vector is

Feeding back to the full connection layer to complete the classification task of the downstream nodes and realize the semantic analysis of the data;

s223, training and supervising a multi-class node classification task:

training a model for supervising a classification task (including conversion and induction) of the nodes of the multiple classes by using the classification cross entropy as a loss function; end-to-end training of the model using the same training targets; for multi-label and multi-class classification, binary cross entropy is used for each class; a categorical cross entropy loss function is used at each step of the LSTM to improve training performance.

Preferably, the step S3 specifically includes:

s31, decomposing a non-negative matrix:

as shown in the formula (3),

is a basis matrix in the matrix decomposition and,

for incomplete modal data instances

Potential representations in a subspace; each modality as defined in equation (1) thus has the same complete modality data encoding matrix P_cThe sum matrix and the mode even encoding matrix can be coupled to minimize the objective function;

s32, regularizing the local invariant graph:

using invariant graph model to pair learned shared coding matrix P_cPerforming regularized representation to ensure that each modal data is consistent with its geometry in the subspace; by constructing a nearest neighbor graph G for each modality^(v)Describing the local geometry between data points; each data instance in modality v

Is shown as G^(v)One point of (1); w^(v)Is G^(v)The weight adjacency matrix of (a); at W^(v)In

Representing data instances

And

degree of closeness between; the measurement method is as follows formula (4):

wherein the content of the first and second substances,

as an example of data

And

the Euclidean distance between the two parts,

and

respectively represent

And

p nearest neighbor data instances;

s33, incomplete multi-mode deep semantic matching fusion:

adopting a fusion deep learning network and incomplete multi-mode deep semantic matching data to jointly mine the deep semantic matching features of twin data in any mode; the model can be represented by equation (5):

wherein the content of the first and second substances,

for the characteristic output of a modal private deep network, f is a nonlinear activation function, here a Sigmod function, W_v、b_vRespectively corresponding weight matrix and offset vector;

and obtaining a multi-mode deep semantic shared subspace by jointly optimizing the modal private deep learning network, the basis matrix and the consistent coding matrix, and performing fusion analysis on multi-mode data characteristics.

The embodiment of the invention correspondingly provides a multi-source heterogeneous twin data fusion system based on a factory production process, which comprises the following steps:

a sensor data acquisition module;

the data classification and semantic analysis module based on the hierarchical feature aggregation model comprises the following modules:

the time sequence dynamic graph structure construction unit is used for modeling the sensor data into a network topological graph of a sensor data time sequence accumulation process; defining edges between sensor nodes as sequential events, and establishing a time sequence dynamic topological graph structure formed by dynamic processes driven by interactive events between the nodes and neighbors of the nodes;

the data classification and semantic analysis unit combines structural information with time information by adopting a GCN-LSTM structure, embeds nodes into a low-dimensional space by utilizing network embedding, simultaneously captures the structure and the property of the network, and learns different feature aggregators on neighborhoods at different depths by adopting a hierarchical feature aggregation method; and aggregating information of different depths using the LSTM to ensure that the information flows from the higher depth to the node;

based on an incomplete multi-mode depth semantic matching fusion model establishing module, adopting a cross-mode depth semantic matching mechanism, and establishing an incomplete multi-mode depth semantic matching fusion model through multi-layer nonlinear correlation among modal data; the method comprises the following steps: constructing a shared characteristic subspace among the modes to learn the sharing of incomplete multi-mode data; local similarity of data of each mode in a shared subspace is ensured by setting a regularization factor of the invariant graph; and establishing a new objective function to describe the incomplete multi-modal data deep semantic matching model.

Preferably, the time sequence dynamic graph structure constructing unit further constructs a heterogeneous multi-source sensor network data topological graph by the following method:

This node embedding can be expressed as

Each embedded depth-k-node embedding can be expressed as

Preferably, the executing process of the data classifying and semantic parsing unit specifically includes:

(1) and embedding the learning nodes through a hierarchical aggregation framework:

where s (x) gives the sample of nodes in neighborhood set x, AGGREGATE is a feature aggregation similar to GraphSAGE, and K ∈ {1, …, K }, where K is the maximum depth that can be found by the sensor data network(ii) a Learned node embedding

And the last node is embedded in h_vIf v is assigned, then equation (2) holds:

finally, the learned vector is

(2) training and supervising the classification task of the various nodes:

training a model for supervising a classification task (including conversion and induction) of the nodes of the multiple classes by using the classification cross entropy as a loss function; end-to-end training of the model using the same training targets; for multi-label and multi-class classification, using binary cross entropy for each class; a categorical cross entropy loss function is used at each step of the LSTM to improve training performance.

Preferably, the executing process based on the incomplete multi-modal deep semantic matching fusion model establishing module specifically includes:

(1) non-negative matrix factorization:

as shown in the formula (3),

is a basis matrix in the matrix decomposition and,

for incomplete modal data instances

(2) and regularizing a local invariant graph:

using invariant graph model to pair learned shared coding matrix P_cPerforming regularization representation to ensure that each modal data is consistent with its geometry in the subspace; by constructing a nearest neighbor graph G for each modality^(v)Describing the local geometry between data points; each data instance in modality v

Is shown as G^(v)One point of (1); w^(v)Is G^(v)The weight adjacency matrix of (a); at W^(v)In (1)

Representing data instances

And

degree of closeness between; the measurement method is as follows formula (4):

wherein the content of the first and second substances,

as an example of data

And

the Euclidean distance between the two parts,

and

respectively represent

And

p nearest neighbor data instances;

(3) incomplete multi-mode deep semantic matching fusion:

wherein the content of the first and second substances,

Another embodiment of the present invention correspondingly provides an apparatus, which includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the multi-source heterogeneous twin data fusion method based on a factory production process according to any one of the above embodiments.

Another embodiment of the present invention correspondingly provides a computer-readable storage medium, where the computer-readable storage medium includes a stored computer program, where the computer program is executed to perform the multi-source heterogeneous twin data fusion method based on a factory production process according to any one of the above embodiments.

Compared with the prior art, the multi-source heterogeneous twin data fusion method and system based on the factory production process provided by the embodiment of the invention are used for modeling sensor data of a plurality of sensors in different production processes of a factory as a network topological graph of a sensor data time sequence accumulation process; defining edges between sensor nodes as sequential events, and establishing a time sequence dynamic topological graph structure formed by dynamic processes driven by interactive events between the nodes and neighbors of the nodes; according to the characteristics of the time sequence dynamic topological graph structure, a GCN-LSTM structure is adopted to combine structural information with time information, nodes are embedded into a low-dimensional space by utilizing network embedding, meanwhile, the structure and the property of the network are captured, and a hierarchical feature aggregation method is adopted to learn different feature aggregators in neighborhoods at different depths; and aggregating information of different depths using the LSTM to ensure that the information flows from the higher depth to the node; and a cross-modal deep semantic matching mechanism is adopted, and an incomplete multi-modal deep semantic matching fusion model is established through the multi-layer nonlinear correlation among modal data. Therefore, the problems that twin data fusion is unbalanced and the like caused by semantic deletion, incomplete mode and unbalanced distribution of multi-source heterogeneous twin data in the production process of an intelligent factory can be effectively solved, heterogeneous twin data of a plurality of sensors in different production processes of the factory are effectively fused, the requirement of unified service is output for upper-layer users, and the effectiveness and the accuracy of fusion between multi-source heterogeneous twin data fusion in the production process of the factory can be improved to a certain extent.

Drawings

In order to more clearly illustrate the technical solution of the present invention, the drawings needed to be used in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without creative efforts.

Fig. 1 is a schematic flow diagram of a multi-source heterogeneous twin data fusion method based on a factory production process according to an embodiment of the present invention.

FIG. 2 is a block diagram of a hierarchical feature aggregation method in a multi-source heterogeneous twin data fusion method based on a factory production process.

FIG. 3 is a flow chart illustrating step S22 of a multi-source heterogeneous twin data fusion method based on a factory production process.

FIG. 4 is an incomplete multi-modal deep semantic matching fusion model in a multi-source heterogeneous twin data fusion method based on a factory production process.

FIG. 5 is a block diagram of a preferred embodiment of a multi-source heterogeneous twin data fusion system based on a plant production process according to the present invention.

FIG. 6 is a block diagram of a preferred embodiment of the data classification and semantic parsing module based on the hierarchical feature aggregation model in FIG. 5.

Fig. 7 is a schematic flow chart of a preferred embodiment of the decision control method based on the 5G-driven intelligent plant digital twin information provided by the invention.

Fig. 8 and 9 are schematic flow charts of the decision control method based on the intelligent plant digital twin information driven by 5G provided by the invention.

Fig. 10 is a flowchart illustrating a preferred embodiment of the 5G-based digital twin information interaction communication flow shown in fig. 7.

Fig. 11 is a 5G-based digital twin communication system framework involved in the 5G-based digital twin information interaction communication flow shown in fig. 7.

Fig. 12 is a schematic structural diagram of a 5G network slice involved in the 5G-based digital twin information interaction communication flow shown in fig. 7.

Fig. 13 is a schematic structural diagram of an SDN involved in the 5G-based digital twin information interaction communication flow shown in fig. 7.

Fig. 14 is a block diagram of edge calculation involved in the 5G-based digital twin information interaction communication flow shown in fig. 7.

FIG. 15 is a schematic diagram of the structure of the intelligent plant edge calculation function involved in the 5G-based digital twin information interaction communication flow shown in FIG. 7.

Fig. 16 is a block diagram of an MEC-based MEC communication system involved in the 5G-based digital twin information interaction communication flow shown in fig. 7.

Fig. 17 is a schematic structural diagram of a digital twin-based deep neural network DNN model involved in the 5G-based digital twin information interaction communication flow shown in fig. 7.

FIG. 18 is a flow diagram of a preferred embodiment of a control decision flow for the multi-source heterogeneous twin data information shown in FIG. 7.

Fig. 19 is a schematic diagram of a deep reinforcement learning competition network architecture involved in a control decision flow of multi-source heterogeneous twin data information shown in fig. 7.

Fig. 20 is a schematic diagram of a neural network architecture involved in the self-supervised multi-modal characterization learning in the control decision flow of multi-source heterogeneous twin data information shown in fig. 7.

FIG. 21 is a schematic flow diagram of a preferred embodiment of a 5G-driven intelligent plant digital twin information-based decision control system provided by the present invention.

Fig. 22 is a schematic structural view of a preferred embodiment of the apparatus provided by the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

It should be noted that the following embodiments may be combined with each other, and the description of the same or similar contents in different embodiments is not repeated.

Referring to fig. 1, a schematic flow chart of a preferred embodiment of a multi-source heterogeneous twin data fusion method based on a factory production process provided by the present invention includes steps S1 to S3, which are as follows:

In step S2, the GCN-LSTM structure is used to combine the structure information with the time information, and the nodes are embedded into the low-dimensional space by using network embedding, and meanwhile, the structure and properties of the network are captured, so as to build a dynamic time-series graph convolution neural network model for the topological graph of the sensor data, where the basic framework of the model is as shown in fig. 2.

Specifically, the step S21 further constructs a heterogeneous multi-source sensor network data topology map in the following manner:

It is known that this node embedding can be expressed as

Each embedded depth-k-node embedding can be expressed as

Further, referring to fig. 3, a schematic flow chart of a preferred embodiment of step S22 in the multi-source heterogeneous twin data fusion method based on the plant production process provided by the present invention includes steps S222 to S223, which are as follows:

And the last node is embedded in h_vIf v is assigned, then equation (2) holds:

finally, the learned vector is

Feeding back to the full connection layer, completing classification tasks of downstream nodes and realizing semantic analysis of data;

s223, training and supervising a multi-class node classification task:

Further, in step S3, an incomplete multimodal data fusion algorithm based on deep semantic matching is adopted, and a unified deep learning model that fuses a modality private deep network and modality sharing features is designed by using correlation of multimodal high-level semantics, so as to implement deep correlation fusion of incomplete multimodal data and reduce semantic deviation of modality sharing features. The algorithm flow of the incomplete multi-modal deep semantic matching fusion model is shown in fig. 4. Specifically, the step S3 specifically includes:

s31, non-negative matrix factorization:

as shown in the formula (3), the,

is a basis matrix in the matrix decomposition and,

for incomplete modal data instances

s32, regularizing the local invariant graph:

using invariant graph model to pair learned shared coding matrix P_cPerforming regular expression to ensure each timeThe individual modality data is consistent with its geometry in the subspace; by constructing a nearest neighbor graph G for each modality^(v)Describing the local geometry between data points; each data instance in modality v

Representing data instances

And

degree of closeness between; the measurement method is as follows formula (4):

wherein the content of the first and second substances,

as an example of data

And

the Euclidean distance between the two parts,

and

respectively represent

And

p nearest neighbor data instances;

s33, incomplete multi-mode deep semantic matching fusion:

wherein the content of the first and second substances,

Referring to fig. 5 and fig. 6, a block diagram of a preferred embodiment of a multi-source heterogeneous twin data fusion system based on a factory production process is provided, which includes the following three modules, respectively:

a sensor data acquisition module 51;

a data classification and semantic parsing module 52 based on a hierarchical feature aggregation model, comprising:

a time sequence dynamic graph structure construction unit 521, which models the sensor data into a network topology graph of a sensor data time sequence accumulation process; defining edges between sensor nodes as sequential events, and establishing a time sequence dynamic topological graph structure formed by dynamic processes driven by interactive events between the nodes and neighbors of the nodes;

the data classification and semantic analysis unit 522 combines structural information and time information by adopting a GCN-LSTM structure, embeds nodes into a low-dimensional space by utilizing network embedding, simultaneously captures the structure and the properties of a network, and learns different feature aggregators on neighborhoods at different depths by adopting a hierarchical feature aggregation method; and aggregating information of different depths using the LSTM to ensure that the information flows from the higher depth to the node;

based on the incomplete multi-mode depth semantic matching fusion model establishing module 53, a cross-mode depth semantic matching mechanism is adopted, and through the multi-layer nonlinear correlation among modal data, an incomplete multi-mode depth semantic matching fusion model is established; the method comprises the following steps: constructing a shared characteristic subspace among the modes to learn the sharing of incomplete multi-mode data; local similarity of data of each mode in a shared subspace is ensured by setting a regularization factor of the invariant graph; and establishing a new objective function to describe the incomplete multi-modal data deep semantic matching model.

In the data classification and semantic analysis module 52 based on the hierarchical feature aggregation model, a GCN-LSTM structure is adopted to combine structural information with time information, nodes are embedded into a low-dimensional space by using network embedding, the structure and the properties of the network are captured at the same time, and a dynamic time-series graph convolution neural network model is established for a topological graph of sensor data, wherein a basic framework of the model is shown in fig. 2.

Specifically, the time sequence dynamic graph structure constructing unit 521 further constructs a heterogeneous multi-source sensor network data topological graph in the following manner:

It is known that this node embedding can be expressed as

Each embedded depth-k-node embedding can be expressed as

Further, the executing process of the data classification and semantic analysis unit specifically includes:

And the last node is embedded in h_vIf v is assigned, then equation (2) holds:

finally, will learnLearned vector

(2) training and supervising the classification task of the various nodes:

Further, in the incomplete multi-modal based depth semantic matching fusion model establishing module 53, an incomplete multi-modal data fusion algorithm based on depth semantic matching is adopted, and a unified deep learning model fusing a modal private depth network and modal sharing features is designed by using correlation of multi-modal high-level semantics, so as to realize depth-related fusion of incomplete multi-modal data and reduce semantic deviation of modal sharing features. The algorithm flow of the incomplete multi-modal deep semantic matching fusion model is shown in fig. 4. The execution process based on the incomplete multi-modal deep semantic matching fusion model establishing module 53 specifically includes:

(1) non-negative matrix factorization:

as shown in the formula (3),

is a basis matrix in the matrix decomposition and,

for incomplete modal data instances

(2) and regularizing a local invariant graph:

Representing data instances

And

degree of closeness between; the measurement method is as follows in formula (4):

wherein the content of the first and second substances,

as an example of data

And

the Euclidean distance between the two parts,

and

respectively represent

And

p nearest neighbor data instances;

(3) incomplete multi-mode deep semantic matching fusion:

wherein the content of the first and second substances,

Referring to fig. 7, a schematic flow chart of a preferred implementation of a decision control method based on intelligent plant digital twin information driven by 5G is provided for an embodiment of the present invention, and includes steps S81 to S83, which are as follows:

s81, fusing multi-source heterogeneous twin data;

s82, carrying out a 5G-based digital twin information interaction communication process; and

and S83, performing control decision flow of multi-source heterogeneous twin data information.

The following describes in detail an execution flow of the decision control method based on the intelligent plant digital twin information driven by 5G according to the present embodiment with reference to fig. 8 and 9.

In step S81, the multi-source isomorphism twin data fusion process adopts the multi-source isomorphism twin data fusion method based on the factory production process according to any of the above embodiments, and details are not repeated here.

Referring to fig. 10, in the step S82, the 5G-based digital twin information interaction communication flow further includes the steps of:

s821: establishing a digital twin communication mode in a 5G environment; and

s822: performance analysis of MEC systems in digital twin communications.

In step S821, the digital twin communication method is established in the sensing terminal layer and the communication network layer of the 5G network, and two corresponding functional systems are emphasized: a network interconnection system and a data intercommunication system. The 5G network slicing function is realized through network interconnection: firstly, logically dividing resources and technologies by using a Software Defined Network (SDN) and a virtualization technology (NFV) on the same physical network infrastructure according to different service scenes and service models; and secondly, network function cutting is made, network resources are managed and arranged, and a plurality of independent virtual networks are formed according to different tasks, so that end-to-end transmission of twin data is realized. And (4) establishing data interconnection at a terminal perception layer, wherein the data interconnection is contained in a physical entity part of a field layer in an actual factory, massive sensors are mapped into a virtual entity by digital twin through data acquisition, analysis and processing, and the information model establishment corresponds to the semantic analysis of the heterogeneous multi-source twin data in the step S81 so as to realize the communication of the twin data at an application layer.

Specifically, as shown in fig. 11, in step S821, a digital twin communication method based on 5G is established, and the communication method is composed of a field layer, an edge layer and a cloud computing layer. Wherein, the field layer is connected with field nodes such as sensors, actuators, equipment, control systems, assets and the like by using a 5G network. The edge layer is positioned between the field layer and the cloud computing layer and comprises two main parts, namely an edge node and an edge manager. And the cloud computing layer completes the intelligent decision of global scheduling. Specifically, the method comprises the following steps:

(a) field layer

A 5G network slice is first established. And virtualizing 5G network physical infrastructure resources into a plurality of mutually independent and parallel virtual network slices according to actual factory workshop task requirements by utilizing an SDN and virtualization technology. As shown in fig. 12, virtual resources are divided into network slices, and subnetworks are created as needed.

And secondly, separating a control plane and a forwarding plane of the SDN according to the actual task requirement of the factory. As shown in fig. 13, task scheduling and resource management are performed in the SDN application layer according to different plant tasks of a factory. At the control layer of the SDN, data plane resources are processed, and network states, network topologies, and the like are maintained. And processing and forwarding various twin data at a data forwarding layer, and collecting the device state.

The field node is then connected to the physical device. The field nodes are connected with edge gateways and other equipment in the edge layer through various types of field networks and industrial buses in the factory production environment and 5G network slices, and communication of data flow and control flow between the field layer and the edge layer is achieved.

And finally, carrying out network structure topology. By connecting the sets of field nodes to each other and to the wide area network using devices such as edge gateways, networks implementing communication between different layers may use different topologies and allow data ingress from edge nodes and control command egress to edge nodes.

(b) Edge calculation

As shown in fig. 14, the framework of edge computing is composed of five parts, namely cloud, edge, field device and its related functions, and specific application.

According to the actual production task requirement of the factory and the situation of the field device, the following functions are realized through edge calculation according to the edge calculation framework in fig. 14, and are specifically shown in fig. 15.

Firstly, designing a connecting structure of an edge layer: (1) support access to various field devices down: such as a manipulator, a machine tool, an AGV and various sensors of a production factory, the edge cloud can be flexibly counted in the edge cloud through a field bus or a 5G network slice, intelligent sensing and calculation are realized through a time sensitive network TSN and an SDN, data analysis and real-time control are carried out, or the whole production process is optimized. (2) The function of global scheduling or intelligent decision is realized through the up-down butt joint of the 5G network slices and the cloud.

And secondly, distributing the tasks of the edge computing nodes. According to different production task emphasis points and different production workshop hardware characteristics, specific task allocation of the edge computing nodes is arranged, wherein the specific task allocation comprises an edge network card which is processed and converted into emphasis points by a network protocol, an edge controller which is emphasized to support real-time closed-loop control service, an edge cloud which is emphasized to large-scale data processing, an edge sensor which is emphasized to low-power consumption information acquisition and processing, and the like.

And finally, setting the use mode of the edge computing system to the resources. (1) Directly packaging computing, network and storage resources, providing a calling interface, and using edge node resources by an edge manager in the modes of code downloading, network configuration, database operation and the like; (2) further, the edge basic resources are packaged into function modules according to the function field, and the edge manager combines and calls the function modules in a mode of model-driven service arrangement to realize integrated development and agile deployment of edge computing services.

Further, in step S822, the data communication process, the transmission state, and the quality service condition of the actual plant during the production process are physically embodied, and the performance of the communication system is analyzed and tested in the virtual entity of the twin mapping according to the established digital twin communication mode in the 5G environment, so as to solve the problems of congestion, delay, and poor quality of the twin data communication. By establishing an MEC system with ultra-reliable and low-delay communication services and delay tolerant services, Energy Efficiency (EE) of users in the MEC system is improved under the condition of meeting the delay and reliability constraints of URLLC services and the stability constraints of the delay tolerant services. The communication network construction of optimal resource allocation is realized by training a Deep Learning (DL) network structure through a digital twin body by adopting measurement parameters of a real network of an actual factory.

Specifically, in step S822, after the 5G-based digital twin communication method is established, the interaction and communication of the twin data depend on the performance of the 5G network slice and the MEC, and the performance simulation analysis is performed for both the business service and the system loss. The method specifically comprises the following steps:

(a) establishing a MEC system communication network

As shown in fig. 16, by establishing an MEC communication system with two types of service traffic, i.e., URLLC service subscribers and delay tolerant service subscribers, the delay, reliability constraints, and stability constraints of the delay tolerant service are met.

URLLC service users and delay tolerant service users are established. M access points AP are K^uIndividual URLLC users and K^bA delay tolerant user providing service, wherein K^uAnd K^bRespectively take on the value of kappa^u＝{1,…,κ^uAnd k^b＝{K^u+1,…,K^u+K^b}. Different service users are distinguished by superscript xi ═ { u, b }. If xi ═ u, this parameter is indicated for URLLC users. Otherwise, for delay tolerant users.

Different attachment points AP are connected to the MME responsible for user association. To establish the digital twin, the MME sends the parameters and model of the network to a central server and explores the user association scheme of the digital twin. The output value of a deep neural network DNN is used to approximate the optimal user association scheme, wherein the DNN is trained offline in the digital twin. And after the training phase is finished, sending a DNN output result associated with one user to the MME. The network is decomposed into a single AP problem according to a given user association scheme. For each single AP problem, the AP optimizes resource allocation and load task transfer for the users associated therewith.

The MEC server is matched for each AP and each user is linked to a local server. The time is discretized into time slots. Each time slot having a duration of T_s. The service efficiency of the mth MEC and the kth user can be respectively expressed as S_m(CPU cycles/slots) and

(CPU cycles/slot). The kth user may be at

In-range adjustment

Is the maximum computing power of the user.

Non-stationary parameters in the system are divided into two categories. The first category of parameters is highly dynamic, such as large-scale channel gain and average task arrival rate. Another type of parameter varies slowly, such as the user density in a certain area. For the first type of parameters, it is directly used as input for DNN. For the second category of parameters, the system detects their values in real time and updates them in the digital twin. The DNN is then learned from the updated digital twin. Instead of training the new DNN network from scratch, the new DNN network is initialized with a well pre-trained DNN. In this way, the output of the DNN will vary as the non-stationary parameters vary.

(b) DNN model analysis of MEC system communication performance parameters

The MEC communication system performance in fig. 16 is analyzed by using a DNN network and a digital twin, and a specific method is shown in fig. 17, and includes:

first, the normalized energy of the user connection method is calculatedThe amount is lost. The input of the deep neural network DNN is the user wide-range channel gain of the connection access point and the user's average mission achievement rate, and the DNN output is the user's connection scheme. Defining the direct output of DNN as

The normalized energy loss of the user connection scheme can be derived from the digital twin model.

And secondly to minimize the normalized energy loss. The minimum normalized energy consumption is solved for the user connection scheme randomly generated according to the exploration strategy, and the optimized and updated optimal user connection scheme is obtained through feedback from the digital twin model

Then, the input and output parameters of the DNN are saved. For the parameters alpha, lambda of the input DNN and the optimal output

And saving the DNN training data to a memory for next DNN training.

And finally, establishing a rule for normalizing energy loss. The digital twin in fig. 17 has system parameters iteratively output via the DNN network, a twin model, and rules set for the access point AP, as shown in table 1.

TABLE 1 normalized energy loss rule

Wherein, P represents a strategy for optimizing channel subcarrier allocation and load probability, and is applicable to all user situations, and the optimization rule is shown as formula (6).

Wherein，

Represents the standard energy loss, interval (N)^*,x^*) The optimum relationship between (α, λ, β) and (α, λ, β) can be expressed as π₂：α，λ，β→N^*，x^*。π₂Representing the minimum normalized energy loss, which can be expressed as Q^*(α，λ，β|π₂) Indicating that the normalized energy loss depends on the user connection.

Referring to fig. 18, in the step S83, the flow of control decision of the multi-source heterogeneous twin data information further includes the steps of:

s831, establishing a decision method based on a deep reinforcement learning competition network architecture; and

and S832, self-supervision learning of multi-modal expression of tasks in the physical entity.

In step S831, in the digital twin virtual entity, the input data comes from the mass sensors in each production process of the physical entity, and the data modal distribution is unbalanced and incomplete in different management production processes, so that the data is of great high dimension. And performing control strategy learning on the high-dimensional input of the heterogeneous multi-mode in the digital twin virtual entity, and researching a control decision scheme of production and management operation of internal autonomous circulation. A neural Network architecture without Model Free reinforcement learning is designed, and the competition Network (Dual Network) has two independent assessment quantities: one for the state cost function V(s) and one for the state dependent action dominance function A. The method can perform inductive learning among actions (actions) under the condition that no change is imposed on a low-level reinforcement learning algorithm, so as to further achieve the decision-making purpose.

Specifically, aiming at the problem that multi-source heterogeneous data are similar in a digital twin information system in an intelligent factory, a method based on a separate modeling state value and an action advantage function is provided, and an optional framework for a Deep Q Network (DQN) and a related learning scheme are established. The method specifically comprises the following steps:

(a) the network structure is as follows:

as shown in fig. 19, for the structure of Dueling DQN, the full connection layer in DQN is split into two parts, one outputs the value about the state with scalar V, and the other outputs the value about the policy cost function a, i.e.: two data flows ahead of the portion indicated by the arrow in fig. 19. Finally, the Q cost function is partially synthesized as indicated by the arrow. Where V has only one dimension, representing the score for that state, and a is consistent with the dimension of the decision, representing the extra score that can be achieved in relation to that state to perform a certain decision. And then V and A are calculated by a formula to obtain the original meaning.

(b) Combination of V and A:

since the expectation of the policy cost function A is 0, A in the network is actually the average of each A minus all A, so the dominance function A, and the reward function Q are redefined, as shown in equation (7)

A^π(s,a)＝Q^π(s,a)-V^π(s) (7)

Where V(s) is a cost function indicating how well the state is. The Q function indicates the value of a certain decision determined in the current state, and the A function indicates the relative goodness of each action in the current state. (c) Difference function of fixed V and A

Since v(s) is a scalar, the value can be left or right biased in the network without affecting the resulting Q value. Therefore, the values of a and V cannot be directly resolved by the Q value. Since the dulling DQN is an end-to-end training network, there is no separate training V or a cost function. For the network architecture, in the end-to-end training, there is a constant difference between the V and a values, and in order to avoid the variance of the difference value, the deviation function is fixed as shown in equation (8):

this part is intended to be implemented in the network architecture of fig. 19 as indicated by the arrows, requiring a unified standard of evaluation of all states and actions.

Further, in step S832, for a special case of an unstructured environment in the object entity, the operation task requiring control usually requires multi-source heterogeneous sensor acquisition. Due to sample complexity, compact and multi-modal representations of network input data can be learned with self-supervision in physical entities to improve sample efficiency of policy learning. To improve sample efficiency, a neural network-based multi-sensor data feature representation is first learned. The resulting compressed feature vector is then used as an input to a strategy learned through reinforcement learning. And finally, training the representation model through self supervision.

In particular, the value of fusing multi-sensor information and the ability of multi-modal representations in cross-task delivery are evaluated with the goal of learning the strategies that the controller performs the tasks that require manipulation. The method specifically comprises the following steps:

(a) model-free reinforcement learning modeling of task operations

The operations with different tasks in the factory production process are approximated as a model-free reinforcement learning problem, and the performance of the model-free reinforcement learning problem is analyzed under the conditions of dependence on multi-modal feedback and uncertain geometry, clearance and configuration. The selection of a model-free model eliminates the need for an accurate kinetic model, and can be an idealized approximation of the actual production environment and task.

Modeling the operation task as a discrete Markov decision process M with finite time, state space S, action space A, state transition dynamics T: S × A → S, initial state distribution ρ 0, return function R: S × A → R, time T, discount coefficient γ ∈ (0,1), and in order to determine the optimal random strategy π: S → P (A), it is desired to maximize the expected discount reward as shown in equation (9):

(b) network architecture

The architecture of a neural network for multi-modal characterization learning using self-supervision using data from a number of different sensors as model inputs (visual sensors, moment sensors, speed sensors, for example) is shown in fig. 20. And analyzing a learning method of the driving neural network based on reinforcement learning, and applying a control strategy fitted by the trained driving neural network to an actual physical system for verification.

Referring to fig. 21, a block diagram of a preferred embodiment of a decision control system based on 5G-driven intelligent plant digital twin information provided by the present invention includes the following:

211. a multi-source heterogeneous twin data fusion module;

212. a 5G-based digital twin information interactive communication module; and

213. and a control decision module of multi-source heterogeneous twin data information.

The decision control system for intelligent plant digital twin information based on 5G driving provided by the embodiment of the present invention can implement all the processes of the decision control method for intelligent plant digital twin information based on 5G driving described in any of the above embodiments, and the functions and implemented technical effects of each module and unit in the device are respectively the same as those of the block chain based privacy protection asynchronous federation sharing method described in the above embodiments, and are not described herein again.

Therefore, the decision control method and system for digital twin information of the intelligent factory based on 5G driving provided by the embodiment of the invention can effectively solve the problems of twin data fusion imbalance and the like caused by semantic deletion, incomplete mode and unbalanced distribution of multi-source heterogeneous twin data in the production process of the intelligent factory, effectively fuse heterogeneous twin data of a plurality of sensors in different production processes of the factory, further output the demand of unified service for upper-layer users, and improve the effectiveness and accuracy of fusion between multi-source heterogeneous twin data fusion in the production process of the factory to a certain extent; and a 5G-based digital twin communication mode can be established, real-time twin data-driven online simulation is realized, virtual-real mapping and interaction are really realized, the instantaneity of reaching a terminal by decision and control and the autonomy and intellectualization of reaction of the terminal and application are ensured, the upstream and downstream cooperative cooperation of a manufacturing industry industrial chain is promoted, the compact and multi-modal representation of data in an input network is learned by using a reinforcement learning and self-supervision learning method in an actual production environment, the decision and control efficiency of resource allocation, production planning, real-time scheduling, supply chain, logistics and the like is improved, and the production control under dynamic factor disturbance is adapted.

The invention also provides equipment.

As shown in fig. 22, a schematic structural diagram of a preferred embodiment of the apparatus provided by the present invention includes a processor 61, a memory 62, and a computer program stored in the memory 62 and configured to be executed by the processor 61, where when the processor 61 executes the computer program, the multi-source heterogeneous twin data fusion method based on a plant production process or the decision control method based on intelligent plant digital twin information driven by 5G as described in any of the above embodiments is implemented.

It should be noted that fig. 22 only illustrates an example in which one memory and one processor in the device are connected, and in some specific embodiments, the device may further include a plurality of memories and/or a plurality of processors, and the specific number and the connection mode thereof may be set and adapted according to actual needs.

The invention also provides a computer-readable storage medium, which specifically includes a stored computer program, wherein when the computer program runs, a device where the computer-readable storage medium is located is controlled to execute the multi-source heterogeneous twin data fusion method based on the plant production process or the decision control method based on the intelligent plant digital twin information driven by 5G according to any of the above embodiments.

It should be noted that, all or part of the flow in the method according to the above embodiments of the present invention may also be implemented by a computer program instructing related hardware, where the computer program may be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of the above embodiments of the method may be implemented. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U.S. disk, removable hard disk, magnetic diskette, optical disk, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier wave signal, telecommunications signal, and software distribution medium, etc. It should be further noted that the computer readable medium may contain content that is subject to appropriate increase or decrease as required by legislation and patent practice in jurisdictions, for example, in some jurisdictions, computer readable media does not include electrical carrier signals and telecommunications signals in accordance with legislation and patent practice.

While the invention has been described with reference to a preferred embodiment, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.

Claims

1. A multi-source heterogeneous twin data fusion method based on a factory production process is characterized by comprising the following steps:

2. The factory production process-based multi-source heterogeneous twin data fusion method according to claim 1, wherein the step S21 is further used for constructing a heterogeneous multi-source sensor network data topology map by:

It is known that this node embedding can be expressed as

Each embedded depth-k-node embedding can be expressed as

3. The multi-source heterogeneous twin data fusion method based on factory production process according to claim 2, wherein the step S22 specifically includes:

And the last node is embedded in h_vIf v is assigned, then equation (2) holds:

finally, the learned vector is

s223, training and supervising a multi-class node classification task:

4. The multi-source heterogeneous twin data fusion method based on factory production process according to claim 3, wherein the step S3 specifically includes:

s31, decomposing a non-negative matrix:

as shown in the formula (3),

is a basis matrix in the matrix decomposition and,

for incomplete modal data instances

s32, regularizing the local invariant graph:

using invariant graph model to pair learned shared coding matrix P_cPerforming regularization representation to ensure that each modal data is consistent with its geometry in the subspace; by constructing a nearest neighbor graph G for each modality^(v)Describing the local geometry between data points;each data instance in modality v

Is shown as G^(v)One point of (1); w is a group of^(v)Is G^(v)The weight adjacency matrix of (a); at W^(v)In

Representing data instances

And

degree of closeness between; the measurement method is as follows formula (4):

wherein the content of the first and second substances,

as an example of data

And

the Euclidean distance between the two parts,

and

respectively represent

And

p nearest neighbor data instances;

s33, incomplete multi-mode deep semantic matching fusion:

wherein the content of the first and second substances,

5. A multi-source heterogeneous twin data fusion system based on a factory production process, comprising:

a sensor data acquisition module;

the data classification and semantic analysis module based on the hierarchical feature aggregation model comprises:

6. The factory production process-based multi-source heterogeneous twin data fusion system according to claim 5, wherein the time sequence dynamic graph structure construction unit further constructs a heterogeneous multi-source sensor network data topology graph by:

It is known that this node embedding can be expressed as

Each embedded depth-k-node embedding can be expressed as

7. The multi-source heterogeneous twin data fusion system based on factory production process according to claim 6, wherein the execution process of the data classification and semantic analysis unit specifically comprises:

And the last node is embedded in h_vIf v is assigned, then equation (2) holds:

finally, the learned vector is

(2) training and supervising the classification task of the various nodes:

training a model for supervising a classification task (including conversion and induction) of the nodes of the multiple classes by using the classification cross entropy as a loss function; end-to-end training of the model using the same training targets; for multi-label and multi-class classification, using binary cross entropy for each class; a classification cross entropy loss function is used at each step of the LSTM to improve the training effect.

8. The factory production process-based multi-source heterogeneous twin data fusion system according to claim 7, wherein the execution process of the incomplete multi-modal deep semantic matching fusion model building module specifically comprises:

(1) non-negative matrix factorization:

as shown in the formula (3),

is a basis matrix in the matrix decomposition and,

for incomplete modal data instances

Potential representations in a subspace; each mode shape defined in the formula (1) as suchHaving the same complete modal data encoding matrix P_cThe sum matrix and the mode even encoding matrix can be coupled to minimize the objective function;

(2) and regularizing a local invariant graph:

Representing data instances

And

degree of closeness between; the measurement method is as follows formula (4):

wherein the content of the first and second substances,

as an example of data

And

the Euclidean distance between the two parts,

and

respectively represent

And

p nearest neighbor data instances;

(3) incomplete multi-mode deep semantic matching fusion:

wherein the content of the first and second substances,

9. An apparatus comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the plant production process-based multi-source heterogeneous twin data fusion method of any of claims 1-4.

10. A computer-readable storage medium comprising a stored computer program, wherein the computer program when executed performs the multi-source heterogeneous twin data fusion method based on a plant production process of any one of claims 1 to 4.