WO2023050229A1

WO2023050229A1 - Industrial data processing method and apparatus, electronic device, and storage medium

Info

Publication number: WO2023050229A1
Application number: PCT/CN2021/121953
Authority: WO
Inventors: 于禾; 王琪; 周文晶; 田德钰
Original assignee: 西门子股份公司; 西门子（中国）有限公司
Priority date: 2021-09-29
Filing date: 2021-09-29
Publication date: 2023-04-06

Abstract

Provided are an industrial data processing method and apparatus, an electronic device, and a storage medium. The industrial data processing method is used for processing data generated in an industrial production process, and comprises: acquiring field data generated in an industrial production process (101); performing quality inspection on the field data to obtain data quality information for indicating the quality of the field data (102); performing data quality improvement on the field data according to the data quality information, to solve a quality problem existing in the field data (103); and storing the field data subjected to the data quality improvement into a pre-created database (104). The amount of data to be processed in an industrial production process can be reduced.

Description

Industrial data processing method, device, electronic device and storage medium

technical field

The present application relates to the technical field of data processing, and in particular to an industrial data processing method, device, electronic equipment and storage medium.

Background technique

With the continuous advancement of the industrial digitalization process, a large amount of data will be generated in the industrial production process. Based on the data, production process control, production plan formulation, production equipment maintenance, etc. can be realized, making industrial production more intelligent. On the basis of improving production efficiency, reduce Labor intensity of personnel. The basis of industrial digitalization is field data collected by field devices, such as field data collected by sensors. Industrial systems include control systems, operating systems, management systems, and cloud computing platforms to monitor and control the production process based on field data. Field data collected by field devices has data loss due to communication interruption, and data quality problems of dirty data due to data redundancy.

At present, after the on-site data of industrial equipment is collected, the on-site data is directly stored in the database. When using the on-site data, the required data is read from the database through the data warehouse technology (Extract-Transform-Load, ETL). After cleaning, load it to the data demand side.

Due to data quality problems such as missing data and dirty data in the collected field data, the collected field data is directly stored in the database, and the field data needs to be processed through ETL every time the field data is used, resulting in industrial production The amount of data processing in the process is relatively large.

Contents of the invention

In view of this, the industrial data processing method, device, electronic equipment and storage medium provided by the present application can reduce the amount of data processing in the industrial production process.

According to the first aspect of the embodiments of the present application, there is provided an industrial data processing method for processing data generated in an industrial production process, the method comprising:

Obtain on-site data generated during industrial production;

performing quality inspection on the on-site data, and obtaining data quality information indicating the quality of the on-site data;

performing data quality improvement processing on the on-site data according to the data quality information, so as to solve quality problems existing in the on-site data;

The on-site data that has undergone data quality improvement processing is stored in a pre-created database.

In a first possible implementation manner, in combination with the first aspect above, the performing quality inspection on the on-site data to obtain data quality information indicating the quality of the on-site data includes:

calculating a data difference feature of the on-site data, the data difference feature being used to indicate a change trend of the on-site data over time;

Acquire the first field information corresponding to each of the first data quality fields of the field data according to at least one preset first data quality field, and the first data quality fields indicate the fields of the field data quality field;

Acquiring data quality information including the data difference feature and each of the first field information.

In a second possible implementation manner, in combination with the first possible implementation manner above, performing data quality improvement processing on the on-site data according to the data quality information includes:

According to the data difference characteristics included in the data quality information, determine whether the field data has abnormal characteristics, and if the field data has the abnormal characteristics, delete the abnormal data corresponding to the abnormal characteristics of the field data ;

According to the heartbeat signal of the data source of the on-site data, it is judged whether there is data missing in the on-site data, and if there is data missing in the on-site data, data enhancement processing is performed on the on-site data according to a preset data enhancement rule, so as to Supplement missing data in said field data;

According to the first data quality field included in the data quality information, it is judged whether the data format of the field data is the target data format, and if the data format of the field data is not the target data format, the field The data format of the data is converted to the target data format.

In a third possible implementation manner, in combination with the first aspect above, the industrial data processing method further includes:

Obtain the first interaction data output by the first data output terminal in the industrial production system through the presentation layer state conversion interface;

Perform semantic search on the fields in the first interaction data, and judge whether the fields in the first interaction data match the pre-created semantic template;

If the fields in the first interaction data do not match the semantic template, perform data quality improvement processing on the first interaction data, so as to solve the quality problems existing in the first interaction data;

Sending the first interaction data that has undergone data quality improvement processing to a first data receiving end in the industrial production system.

In a fourth possible implementation manner, in combination with the above first aspect or any possible implementation manner of the first aspect, the industrial data processing method further includes:

Obtain the second interaction data output by the second data output terminal in the industrial production system through the message queue;

According to at least one preset second data quality field, respectively acquire second field information corresponding to each of the second data quality fields of the second interaction data, where the second data quality field indicates the first Two fields for the quality of the interaction data;

judging whether there is a data quality problem in the second interaction data according to the information in each second field;

If there is a data quality problem in the second interaction data, perform data quality improvement processing on the second interaction data according to information in each of the second fields, so as to delete abnormal data in the second interaction data. Supplement missing data in the second interactive data, and perform data format conversion on the second interactive data;

Sending the second interaction data that has undergone data quality improvement processing to a second data receiving end in the industrial production system.

According to the second aspect of the embodiments of the present application, there is also provided an industrial data processing device for processing data generated during industrial production, the device comprising:

The acquisition module is used to acquire field data generated in the industrial production process;

A detection module, configured to perform quality detection on the on-site data acquired by the acquisition module, and obtain data quality information indicating the quality of the on-site data;

A processing module, configured to perform data quality improvement processing on the on-site data according to the data quality information acquired by the detection module, so as to solve quality problems existing in the on-site data;

An output module, configured to store the on-site data processed by the processing module for data quality improvement into a pre-created database.

According to the third aspect of the embodiments of the present application, there is also provided an electronic device, including: a processor, a memory, a communication interface, and a communication bus, and the processor, the memory, and the communication interface are completed through the communication bus mutual communication;

The memory is used to store at least one executable instruction, and the executable instruction causes the processor to perform operations corresponding to the industrial data processing method provided in the first aspect or any possible implementation manner of the first aspect.

According to the fourth aspect of the embodiments of the present application, there is also provided a computer-readable storage medium, the computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the processing The processor executes the industrial data processing method provided in the first aspect or any possible implementation manner of the first aspect.

According to a fifth aspect of the embodiments of the present application, there is also provided a computer program, including computer-executable instructions. When executed, the computer-executable instructions cause at least one processor to perform the above-mentioned first aspect or the first aspect. An industrial data processing method provided by any possible implementation.

According to a sixth aspect of the embodiments of the present application, there is also provided a computer program product, the computer program product is tangibly stored on a computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed When at least one processor executes the industrial data processing method provided in the first aspect or any possible implementation manner of the first aspect.

It can be seen from the above technical solution that after obtaining the on-site data of the industrial production system, the on-site data is firstly inspected for data quality, and the data quality information indicating the quality of the on-site data is obtained, and then the on-site data is subjected to data quality improvement processing according to the data quality information. Solve the data quality problems existing in the field data, and then store the field data processed by data quality improvement into the pre-created database. Since the on-site data stored in the database is processed through data quality improvement, the data quality problems existing in the on-site data have been solved, and the applications and services in the industrial production system can directly read the required on-site data from the database for use. It is not necessary to process the field data through ETL every time the field data is read, thereby reducing the amount of data processing in the industrial production process.

Description of drawings

FIG. 1 is a flow chart of an industrial data processing method provided in Embodiment 1 of the present application;

FIG. 2 is a flow chart of a method for acquiring data quality information provided in Embodiment 2 of the present application;

FIG. 3 is a flow chart of a data quality improvement processing method provided in Embodiment 3 of the present application;

FIG. 4 is a flowchart of an industrial data processing method provided in Embodiment 4 of the present application;

5 is a flow chart of an industrial data processing method provided in Embodiment 5 of the present application;

FIG. 6 is a schematic diagram of an industrial data processing device provided in Embodiment 6 of the present application;

FIG. 7 is a schematic diagram of an electronic device provided in Embodiment 7 of the present application.

List of reference signs:

101: Obtain field data generated during industrial production

102: Perform quality inspection on on-site data, and obtain data quality information used to indicate the quality of on-site data

103: Perform data quality improvement processing on field data to solve quality problems existing in field data

104: Store field data processed for data quality improvement into a pre-created database

201: Calculate data difference characteristics of field data

202: Obtain the first field information corresponding to each preset first data quality field in the field data respectively

203: Obtain data quality information including data difference features and first field information

301: According to the data difference characteristics included in the data quality information, determine whether there are abnormal characteristics in the field data

302: Delete the abnormal data corresponding to the abnormal characteristics of the field data

303: According to the heartbeat signal of the data source of the field data, determine whether there is data missing in the field data

304: Perform data enhancement processing on the field data according to the preset data enhancement rules

305: According to the first data quality field, determine whether the data format of the field data is the target data format

306: Convert the data format of field data to the target data format

307: Store field data in database

401: Obtain the first interaction data output by the first data output terminal through the presentation layer state transition interface

402: Determine whether the fields in the first interaction match the pre-created semantic template

403: Perform data quality improvement processing on the first interaction data to solve quality problems existing in the first interaction data

404: Send the first interaction data after solving the quality problem to the first data receiving end

501: Obtain the second interaction data output by the second data output terminal in the industrial production system through the message queue

502: Obtain the second field information of the second interaction data corresponding to each preset second data quality field

503: Judging whether there is a data quality problem in the second interactive data according to the information of each second field

504: Perform data quality improvement processing on the second interaction data according to the information in each second field

505: Send the second interaction data after solving the quality problem to the second data receiving end

100: Industrial data processing method 200: Data quality information acquisition method 300: Data quality improvement processing method

400: Industrial data processing method 500: Industrial data processing method 600: Industrial data processing device

700: Electronic equipment 601: Obtaining module 602: Detection module

603: Processing module 604: Output module 702: Processor

704: Communication interface 706: Memory 708: Communication bus

710: Procedure

Detailed ways

As mentioned above, in the current industrial production system, after the field data is collected from the field equipment, the collected field data is directly stored in the database. When the application or service included in the industrial production system needs to use the field data, the Warehouse technology ETL (Extract-Transform-Load), reads the required field data from the database, cleans the read field data and loads it to the application or service that requests the data. In the process of collecting field data, there may be problems of communication interruption and repeated collection of the same data. Communication interruption will cause data loss in the collected field data, and repeated collection of the same data will cause the collected field data to include dirty data. The collected field data is stored in the database, so the field data stored in the database has data quality problems such as missing data and dirty data. Every time an application or service in an industrial production system uses on-site data, it needs to process the on-site data through ETL, resulting in a large amount of data processing in the industrial production process.

In the embodiment of this application, after obtaining the on-site data generated in the industrial production process, the quality of the on-site data is inspected to obtain the data quality information used to indicate the quality of the on-site data, and then the data quality of the on-site data is improved according to the data quality information processing to solve the data quality problems existing in the field data, and then store the field data that has undergone data quality improvement processing into the database. Since the on-site data stored in the database is processed by data quality improvement, the data quality problems existing in the on-site data have been solved. When the application or service in the industrial production system uses the on-site data, it can directly read the required data from the database. There is no need to process the field data through ETL every time the field data is used, which can reduce the amount of data processing in the industrial production process.

The industrial data processing method and device provided in the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

Embodiment one

FIG. 1 is a flow chart of an industrial data processing method 100 provided in Embodiment 1 of the present application, which is used to process data generated in the industrial production process. As shown in FIG. 1 , the industrial data processing method 100 includes the following steps:

Step 101, acquiring on-site data generated in the industrial production process.

In the process of industrial production, industrial equipment in the industrial production system will generate on-site data, and the on-site data is used to indicate the operating status of the industrial equipment. On-site data can be obtained from gateway devices, sensors, on-site applications or on-site systems, etc., and manual on-site operation information can also be obtained as on-site data. For example, on-site data can be order information, temperature information, vibration information, rotational speed information, etc.

When obtaining field data, field data can be obtained through different types of communication protocols, such as through OPC UA (OPC is the abbreviation of OLE for Process Control, OLE technology applied to process control, OLE technology is Object Linking and Embedding, refers to Object connection and embedding, UA is the abbreviation of Unified Architecture, refers to unified architecture), message queue telemetry transmission (Message Queuing Telemetry Transport, MQTT) or hypertext transfer protocol (Hyper Text Transfer Protocol, HTTP) and other communication protocols to obtain field data.

102. Perform quality inspection on the on-site data, and obtain data quality information for indicating the quality of the on-site data.

In the process of obtaining the field data, the data quality of the field data is detected based on the communication protocol for obtaining the field data, and the data quality information for indicating the quality of the field data is obtained. Through the data quality information, it can be determined whether there are data quality problems such as missing data and dirty data in the field data.

The quality inspection of on-site data can obtain on-site data from different data sources through different communication protocols through the software development kit (Software Development Kit, SDK) or embedded program modules, and perform data analysis on the received on-site data. Quality inspection to obtain corresponding data quality information.

Step 103, according to the data quality information, perform data quality improvement processing on the on-site data, so as to solve the quality problems existing in the on-site data.

Since the data quality information can indicate the data quality problems existing in the field data, it is possible to improve the data quality of the field data based on the data quality information to solve the data quality problems existing in the field data. For example, if the data quality information indicates that there is data missing in the on-site data, the missing data in the on-site data can be supplemented according to the data quality information. Dirty data deletion.

Step 104, storing the on-site data processed for data quality improvement into a pre-created database.

By improving the data quality of on-site data and solving the data quality problems existing in the on-site data, the on-site data is stored in the database for use in applications or services in industrial production systems. The application or service in the industrial production system can read the required field data from the database through the data reading interface of the database, so as to perform industrial equipment control, production plan formulation, maintenance plan formulation, etc. based on the field data.

In the embodiment of the present application, after obtaining the on-site data of the industrial production system, first perform data quality inspection on the on-site data, obtain data quality information indicating the quality of the on-site data, and then perform data quality improvement processing on the on-site data according to the data quality information , solve the data quality problems existing in the field data, and then store the field data processed by data quality improvement into the pre-created database. Since the on-site data stored in the database is processed through data quality improvement, the data quality problems existing in the on-site data have been solved, and the applications and services in the industrial production system can directly read the required on-site data from the database for use. It is not necessary to process the field data through ETL every time the field data is read, thereby reducing the amount of data processing in the industrial production process.

Embodiment two

On the basis of the industrial data processing method provided in Embodiment 1, when the data quality information is obtained by performing quality inspection on the on-site data, the change trend of the on-site data over time and some key fields in the on-site data that can reflect the data quality as data quality information.

FIG. 2 is a flowchart of a data quality information acquisition method 200 provided in Embodiment 2 of the present application. As shown in FIG. 2 , the data quality information acquisition method 200 includes the following steps:

Step 201, calculating data difference characteristics of field data.

Data variance features are used to indicate trends in field data over time. For time series data, if there is a sudden change in the data and the data source is running normally, it means that abnormal data is generated due to interference factors. Therefore, according to the data difference characteristics, it can be judged whether the field data includes abnormal data.

The data difference feature may be the variance, mean, and standard deviation of field data, etc., which characterize the parameter values of field data distribution.

Step 202: Obtain first field information corresponding to each first data quality field in the field data according to at least one preset first data quality field.

The first data quality field is a field that can indicate the data quality of the field data in the on-site data. For example, the first data quality field can include data acquisition time, numerical value, data sequence, signal quality, and the last variance value (last D-value ), the number of characters and data bits included, etc.

The first data quality field may be set through configuration information or a user experience (User Experience, UX) service.

It should be noted that, depending on the communication protocol used when acquiring field data, for a communication protocol whose communication field includes a first data quality field, the first field information corresponding to each first data quality field can be obtained directly from the communication field, For example, when obtaining field data through OPC UA, the first field information can be obtained directly from the communication field. For a communication protocol whose communication field does not include a first data quality field, first field information corresponding to each first data quality field may be obtained by detecting and calculating on-site data.

Step 203. Obtain data quality information including data difference features and first field information.

After calculating the data difference characteristics of the field data and obtaining the first field information corresponding to each first data quality field in the field data, the data difference characteristics and each first field information are used as data quality information to indicate the field data data quality.

When receiving field data through the communication protocol provided by the SDK or embedded program module, the data difference characteristics and the first field information can be embedded into the communication field according to the communication protocol used by the SDK or embedded program module, and the field data Send them together to the data quality controller, and then the data quality controller can improve the data quality of the received field data according to the received data quality information.

In the embodiment of this application, the data difference feature can indicate the change trend of the field data over time, so it can be determined according to the data difference feature whether there is abnormal data in the field data that suddenly increases or decreases suddenly, and the first data quality field is the indicator data The key field of quality, so according to the first field information corresponding to the first data field, it can be determined whether there are problems such as data format in the field data. Obtaining data quality information including data difference characteristics and first field information, and then improving the data quality of field data according to data difference characteristics and each first field information, can more comprehensively solve data quality problems existing in field data and ensure The on-site data stored in the database has no or less data quality problems, thereby ensuring that applications and services in the industrial production system can operate normally based on the on-site data in the database.

Embodiment Three

On the basis of the data quality information acquisition method provided in the embodiment, after acquiring the data quality information including the data difference feature and the first field information, the abnormal data problems and data missing problems included in the field data can be analyzed according to the data quality information and format exceptions are handled.

FIG. 3 is a flowchart of a data quality improvement processing method 300 provided in Embodiment 3 of the present application. As shown in FIG. 3 , the data quality improvement processing method 300 includes the following steps:

Step 301 , according to the data difference features included in the data quality information, judge whether there are abnormal features in the field data, if yes, go to step 302 , if not N, go to step 303 .

Since the data difference feature indicates the change trend of the field data over time, it can be judged whether there is a sudden increase or decrease in the field data according to the data difference feature. If there is a sudden increase or decrease in the field data, then If it is determined that there is an abnormal feature in the field data, step 302 is executed accordingly; if there is no sudden increase or decrease in the field data, it is determined that there is no abnormal feature in the field data, and step 303 is executed accordingly.

Anomaly features can be parameters such as variance, mean, and standard deviation of field data that reflect changes in field data over time.

Step 302, deleting the abnormal data corresponding to the abnormal feature of the field data.

According to the abnormal characteristics of the field data, the abnormal data that causes the abnormal characteristics to appear in the field data can be determined. Dirty data problem.

For example, there are abnormal data in the field data that suddenly increase due to human interference. Abnormal data in the field data, and then delete the abnormal data in the field data.

Step 303 , according to the heartbeat signal of the data source of the on-site data, judge whether there is data missing in the on-site data, if yes, go to step 304 , if not N, go to step 305 .

In the process of acquiring field data, the data source of the field data sends a heartbeat signal to determine whether the communication is connected. If the heartbeat signal of the data source of the field data times out, that is, the data source from the field data is not received within the preset signal cycle The heartbeat signal indicates that the communication connection with the data source of the field data has been disconnected, so the field data cannot be obtained from the data source of the field data, resulting in data loss in the field data. If the heartbeat signal from the data source of the field data can be received according to the preset signal cycle, it means that the communication with the data source of the field data remains connected, and the field data can be obtained from the data source of the field data normally, so as to determine There is no missing data in field data.

Step 304, perform data enhancement processing on the field data according to preset data enhancement rules.

After it is determined that there is data missing in the on-site data, according to the preset data enhancement rules, data enhancement processing is performed on the on-site data to generate corresponding data as missing data to supplement the on-site data, so as to solve the problem of missing data in the on-site data.

When performing data enhancement processing on the field data, the missing field data can be supplemented according to the change trend of the field data over time, or the average value of the obtained field data can be calculated, and the calculated average value can be used as the missing field data.

Step 305 , according to the first data quality field included in the data quality information, judge whether the data format of the field data is the target data format, if yes, execute step 307 , if not N, execute step 306 .

The first data quality field is a field that reflects the quality of on-site data. According to the first data quality field, it can be judged whether the data format of the on-site data is the target data format required by the application and service in the industrial production system. If the data format of the on-site data is the target data format, the on-site data can be directly stored in the database, and step 307 is executed accordingly. If the data format of the field data is not the target data format, the data format of the field data needs to be converted, and step 306 is executed accordingly.

Step 306, converting the data format of the field data into the target data format.

After determining that the data format of the field data is not the target data format, the data format of the field data is converted, and the data format of the field data is converted into the target data format.

For example, if the binary digits of on-site data are 8 bits, and the binary digits of the data that can be processed by applications and services in the industrial production system are 16 bits, then the field data is converted to include 16 bits by complementing 0 or 1. Bit-by-bit data to meet the needs of applications or services in industrial production systems.

Step 307, store the field data in the database.

In the embodiment of this application, according to the data quality information, the dirty data included in the field data is deleted, the missing data is supplemented in the field data, and the data format of the field data is converted to the target data required by the application or service in the industrial production system format, which can more comprehensively solve the data quality problems of on-site data, ensure that the on-site data stored in the data block does not exist or has only a few data quality problems, and thus ensures that the applications and services in the industrial production system can directly use the data On-site data in the block, thereby reducing the amount of data processing in the industrial production process.

Embodiment four

In the industrial production system, not only the on-site data has data quality problems, but also the interactive data between applications and applications, applications and services, and services and services has data quality problems. To be able to interact normally, it is necessary to solve the quality problem of the interactive data.

FIG. 4 is a flowchart of an industrial data processing method 400 provided in Embodiment 4 of the present application. As shown in FIG. 4 , the industrial data processing method 400 includes the following steps:

Step 401. Obtain the first interaction data output by the first data output terminal in the industrial production system through the presentation layer state conversion interface.

Applications and services in the industrial production system can interact through the presentation layer state transition interface (Restful interface). An application can interact with other applications or services through the presentation layer state transition interface, and a service can interact with other applications or services through the presentation layer state transition interface. The interface interacts with other services or applications, so the first data output end may be an application or service, and the corresponding first data receiving end may also be an application or service.

When the first data output terminal and the first data receiving terminal interact through the presentation layer state transition interface, the first interaction data output by the first data output terminal is acquired.

Step 402 , perform semantic search on the fields in the first interaction data, and judge whether the fields in the first interaction match the pre-created semantic template, if yes, perform step 404 , if not N, perform step 403 .

Create a semantic template in advance according to the structure of the communication interface between the first data output terminal and the first data receiving terminal, and the field information of the interactive data. The semantic template defines the interaction data between the first data output terminal and the first data receiving terminal. The fields to include, and the data format for each field. After obtaining the first interaction data output by the first data output terminal, perform a semantic search on the first interaction data to determine the fields included in the first interaction data and the data corresponding to each field, if the fields included in the first interaction data are the same as The fields defined by the semantic template are matched one by one, and the data corresponding to each field in the first interaction data satisfies the data format defined by the semantic template, then it is determined that the first interaction data matches the semantic template, and step 404 is executed accordingly. If the fields included in the first interaction data are missing or have more fields than the fields defined in the semantic template, or the data corresponding to one or more fields in the first interaction data does not meet the data format defined in the semantic template, Then it is determined that the first interaction data does not match the semantic template, and step 403 is executed accordingly.

Step 403: Perform data quality improvement processing on the first interaction data, so as to solve quality problems existing in the first interaction data.

When it is determined that the first interaction data does not match the semantic template, it means that there is a data quality problem in the first interaction data, and data quality improvement processing is performed on the first interaction data to solve the data quality problem in the first interaction data.

To improve the data quality of the first interaction data, please refer to the data quality improvement process of the field data in the third embodiment, such as deleting dirty data in the first interaction data, supplementing missing data in the first interaction data, and improving the data quality of the first interaction data. - Interactive data for data format conversion, etc.

Step 404: Send the first interaction data after the quality problem is solved to the first data receiving end.

If it is determined that the first interaction data matches the semantic template, that is, there is no data quality problem in the first interaction data, the first interaction data is directly sent to the first data receiving end. If it is determined that the first interaction data does not match the semantic template, that is, there is a data quality problem in the first interaction data, after performing data quality improvement processing on the first interaction data, the first interaction data that has undergone data quality improvement processing is sent to the first interaction data. A data receiving end.

In this embodiment of the present application, the semantic template is created in advance, and whether the first interaction data matches the semantic template is judged through semantic search, so as to determine whether the first interaction data has data quality problems, and the first interaction data with data quality problems Perform data quality improvement processing, send the first interactive data after solving the data quality problem to the first data receiving end, and ensure that the first data output end and the first data receiving end interacting through the presentation layer state conversion interface can interact normally, so that In the industrial production system, the applications and services that interact through the state transition interface of the presentation layer can normally obtain the interactive data, thereby ensuring the normal operation of the industrial production system and the efficiency of industrial production.

Embodiment five

The industrial data processing method provided by Embodiment 4 is used to solve the quality problem of the interactive data transmitted through the presentation layer state transition interface. The application and service in the industrial production system can also interact through the message queue (Message queue). In order to further improve the industrial The quality of data in the production system can also solve the quality problem of interactive data transmitted through message queues.

Fig. 5 is a flowchart of an industrial data processing method 500 provided in Embodiment 5 of the present application. As shown in Fig. 5, the industrial data processing method 500 includes the following steps:

Step 501, acquire the second interaction data output by the second data output terminal in the industrial production system through the message queue.

Applications and services in industrial production systems can interact through message queues. An application can interact with other applications or services through message queues, and a service can also interact with other services or applications through message queues, so output data The second data output end may be an application or a service, and the corresponding second data receiving end receiving data may also be an application or a service.

When the second data output terminal and the second data receiving terminal interact through the message queue, the second interaction data output by the second data output terminal is acquired.

Step 502: Acquire second field information corresponding to each second data quality field of the second interaction data according to at least one preset second data quality field.

For the second interaction data transmitted through the message queue, at least one second data quality field used to indicate the data quality of the second interaction data is predetermined, and after the second interaction data is obtained, each The second field information corresponding to the second data quality field. The second data quality field may include data acquisition time, value size, data sequence, signal quality, last variance value (last D-value), number of included characters and number of data bits, etc.

The second data quality field can be set through configuration information or user experience (User Experience, UX) service.

Step 503 , judging whether there is a data quality problem in the second interaction data according to the information of each second field, if yes, go to step 504 , if not N, go to step 505 .

Since the second data quality field is a field indicating the quality of the second interaction data, it can be determined whether there is a data quality problem in the second interaction data according to the second field information corresponding to the second data quality field, such as whether the second interaction data is missing a field , The format of the second field information is incorrect, including redundant fields, etc. If there is a data quality problem in the second interaction data, step 504 is executed to deal with the data quality problem in the second interaction data. If there is no data quality problem in the second interaction data, step 505 is performed to directly send the second interaction data to the second data receiving end.

Step 504: Perform data quality improvement processing on the second interaction data according to the information of each second field.

When there is a data quality problem in the second interaction data, perform data quality improvement processing on the second interaction data according to the information of each second field, so as to delete abnormal data in the second interaction data and supplement missing data in the second interaction data, And performing data format conversion on the second interaction data. Deleting the abnormal data in the second interactive data is mainly to delete dirty data such as redundant data in the second interactive data. The data format conversion of the second interaction data is mainly to convert the second field information in the second interaction data into the data format required by the second data receiving end.

Step 505. Send the second interaction data after the quality problem is solved to the second data receiving end.

If it is determined that there is no data quality problem in the second interaction data, the second interaction data is directly sent to the second data receiving end. If it is determined that there is a data quality problem in the second interaction data, after data quality improvement processing is performed on the second interaction data, the second interaction data that has undergone data quality improvement processing is sent to the second data receiving end.

In this embodiment of the present application, the second data quality field used to indicate the quality of the second interaction data is predetermined, and the second field information corresponding to each second data quality field is extracted from the second interaction data, and then according to the The two-field information can perform data quality improvement processing on the second interaction data, and send the second interaction data after solving the data quality problem to the second data receiving end, so as to ensure the second data output end and the second data receiving end through the message queue interaction The terminal can interact normally, so that the applications and services interacting through the message queue in the industrial production system can normally obtain the interactive data, thereby ensuring the normal operation of the industrial production system and the efficiency of industrial production.

Embodiment six

Fig. 6 is a schematic diagram of an industrial data processing device 600 provided in Embodiment 6 of the present application, which is used to process data generated in the industrial production process. As shown in Fig. 6, the industrial data processing device 600 includes:

An acquisition module 601, configured to acquire field data generated during industrial production;

The detection module 602 is configured to perform quality detection on the on-site data acquired by the acquisition module 601, and obtain data quality information indicating the quality of the on-site data;

The processing module 603 is configured to perform data quality improvement processing on the on-site data according to the data quality information acquired by the detection module 602, so as to solve the quality problems existing in the on-site data;

The output module 604 is configured to store the on-site data processed by the processing module 603 for data quality improvement into a pre-created database.

In this embodiment of the present application, the first acquisition module 601 can be used to execute step 101 in the first embodiment above, the detection module 602 can be used to execute step 102 in the first embodiment above, and the processing module 603 can be used to execute the step 102 in the first embodiment above. In step 103, the output module 604 can be used to execute step 104 in the first embodiment above.

In a possible implementation manner, the detection module 602 may be configured to perform the following operations:

Calculate the data difference characteristics of the field data, and the data difference characteristics are used to indicate the change trend of the field data over time;

According to at least one preset first data quality field, first field information corresponding to each first data quality field of the field data is respectively obtained, and the first data quality field is a field indicating the quality of the field data;

Data quality information including data difference features and information of each first field is acquired.

In this embodiment of the present application, the detection module 602 may be used to execute steps 201 to 203 in the second embodiment above.

In a possible implementation manner, the processing module 603 may be configured to perform the following operations:

According to the data difference characteristics included in the data quality information, it is judged whether there are abnormal characteristics in the field data, and if there are abnormal characteristics in the field data, the abnormal data corresponding to the abnormal characteristics of the field data is deleted;

According to the heartbeat signal of the data source of the on-site data, it is judged whether there is data missing in the on-site data. If there is data missing in the on-site data, data enhancement processing is performed on the on-site data according to the preset data enhancement rules to supplement the missing data in the on-site data. ;

According to the first data quality field included in the data quality information, it is judged whether the data format of the field data is the target data format, and if the data format of the field data is not the target data format, then the data format of the field data is converted into the target data format.

In this embodiment of the present application, the processing module 603 may be configured to execute steps 301 to 306 in the third embodiment above.

In a possible implementation, as shown in Figure 6,

The obtaining module 601 is also used to obtain the first interaction data output by the first data output terminal in the industrial production system through the presentation layer state conversion interface;

The detection module 602 is further configured to perform semantic retrieval on the fields in the first interaction data acquired by the acquisition module 601, and determine whether the fields in the first interaction data match the pre-created semantic template;

The processing module 603 is further configured to perform data quality improvement processing on the first interaction data when the detection module 602 determines that the fields in the first interaction data do not match the semantic template, so as to solve the quality problems existing in the first interaction data;

The output module 604 is further configured to send the first interaction data processed by the processing module 603 to improve the data quality to the first data receiving end in the industrial production system.

In this embodiment of the present application, the acquisition module 601 can be used to execute the step 401 in the fourth embodiment above, the detection module 602 can be used to execute the step 402 in the fourth embodiment above, and the processing module 603 can be used to execute the steps in the fourth embodiment above 403. The output module 604 can be used to execute step 404 in the fourth embodiment above.

In a possible implementation, as shown in Figure 6,

The obtaining module 601 is also used to obtain the second interaction data output by the second data output terminal in the industrial production system through the message queue;

The detection module 602 is further configured to respectively acquire second field information corresponding to each second data quality field of the second interaction data according to at least one preset second data quality field, where the second data quality field indicates the second The field of the quality of the interactive data, and judge whether there is a data quality problem in the second interactive data according to the information of each second field;

The processing module 603 is further configured to determine in the detection module 602 that there is a data quality problem in the second interaction data, and perform data quality improvement processing on the second interaction data according to the information of each second field, so as to delete abnormal data in the second interaction data , supplementing missing data in the second interaction data, and performing data format conversion on the second interaction data;

The output module 604 is further configured to send the second interaction data processed by the processing module 603 to improve the data quality to the second data receiving end in the industrial production system.

In this embodiment of the application, the acquiring module 601 can be used to execute step 501 in the fifth embodiment above, the detection module 602 can be used to execute step 502 and step 503 in the fifth embodiment above, and the processing module 603 can be used to execute the fifth embodiment above In step 504, the output module 604 can be used to execute step 505 in the fifth embodiment above.

It should be noted that the interaction between the various modules in the above-mentioned device embodiment is based on the same inventive concept as that of the foregoing method embodiment, and details can be found in the description of the foregoing method embodiment, which will not be repeated here.

Embodiment seven

FIG. 7 is a schematic diagram of an electronic device provided in Embodiment 7 of the present application. The specific embodiment of the present application does not limit the specific implementation of the electronic device. Referring to FIG. 7 , an electronic device 700 provided by an embodiment of the present application includes: a processor (processor) 702 , a communication interface (Communications Interface) 704 , a memory (memory) 706 , and a communication bus 708 . in:

The processor 702 , the communication interface 704 , and the memory 706 communicate with each other through the communication bus 708 .

The communication interface 704 is used for communicating with other electronic devices or servers.

The processor 702 is configured to execute the program 710, specifically, may execute relevant steps in the above-mentioned embodiments of the industrial data processing method.

Specifically, the program 710 may include program codes including computer operation instructions.

The processor 702 may be a central processing unit CPU, or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be of the same type, such as one or more CPUs, or may be different types of processors, such as one or more CPUs and one or more ASICs.

The memory 706 is used for storing the program 710 . The memory 706 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

The program 710 may specifically be used to enable the processor 702 to execute the industrial data processing method in any of the foregoing embodiments.

For the specific implementation of each step in the program 710, reference may be made to the corresponding description of the corresponding steps and units in the above-mentioned embodiment of the industrial data processing method, and details are not repeated here. Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the foregoing method embodiments, and details are not repeated here.

Through the electronic device of this embodiment, after the on-site data of the industrial production system is obtained, the data quality inspection is first performed on the on-site data, and the data quality information indicating the quality of the on-site data is obtained, and then the data quality of the on-site data is improved according to the data quality information Processing to solve the data quality problems existing in the field data, and then store the field data processed by data quality improvement into the pre-created database. Since the on-site data stored in the database is processed through data quality improvement, the data quality problems existing in the on-site data have been solved, and the applications and services in the industrial production system can directly read the required on-site data from the database for use. It is not necessary to process the field data through ETL every time the field data is read, thereby reducing the amount of data processing in the industrial production process.

The present application also provides a computer-readable storage medium storing instructions for causing a machine to execute the industrial data processing method as described herein. Specifically, a system or device equipped with a storage medium may be provided, on which a software program code for realizing the functions of any of the above embodiments is stored, and the computer (or CPU or MPU of the system or device) ) to read and execute the program code stored in the storage medium.

In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, so the program code and the storage medium storing the program code constitute a part of the present application.

Examples of storage media for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), Tape, non-volatile memory card, and ROM. Alternatively, the program code can be downloaded from a server computer via a communication network.

In addition, it should be clear that not only by executing the program code read by the computer, but also by making the operating system on the computer complete part or all of the actual operations through instructions based on the program code, so as to realize the function of any one of the embodiments.

In addition, it can be understood that the program code read from the storage medium is written into the memory provided in the expansion board inserted into the computer or written into the memory provided in the expansion module connected to the computer, and then based on the program code The instruction causes the CPU installed on the expansion board or the expansion module to perform some or all of the actual operations, thereby realizing the functions of any one of the above-mentioned embodiments.

The embodiment of the present application also provides a computer program, including computer executable instructions, when executed, the computer executable instructions cause at least one processor to execute the industrial data processing methods provided in the above embodiments.

The embodiment of the present application also provides a computer program product, the computer program product is tangibly stored on a computer-readable medium and includes computer-executable instructions, and the computer-executable instructions cause at least one processor to Execute the industrial data processing methods provided by the above-mentioned embodiments. It should be understood that the solutions in this embodiment have the corresponding technical effects in the foregoing method embodiments, and details are not repeated here.

It should be noted that not all the steps and modules in the above flow charts and device structure diagrams are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as required. The system structures described in the above embodiments may be physical structures or logical structures, that is, some modules may be realized by the same physical entity, or some modules may be realized by multiple physical entities, or may be realized by multiple Certain components in individual devices are implemented together.

In the above embodiments, the hardware modules may be implemented mechanically or electrically. For example, a hardware module may include permanently dedicated circuitry or logic (such as a dedicated processor, FPGA or ASIC) to perform the corresponding operations. The hardware modules may also include programmable logic or circuits (such as general-purpose processors or other programmable processors), which can be temporarily set by software to complete corresponding operations. The specific implementation (mechanical way, or a dedicated permanent circuit, or a temporary circuit) can be determined based on cost and time considerations.

The above has shown and described the application in detail through the accompanying drawings and preferred embodiments, but the application is not limited to these disclosed embodiments, and those skilled in the art based on the above-mentioned multiple embodiments can know that the above-mentioned different embodiments can be combined The code auditing means in the present application obtains more embodiments, and these embodiments are also within the protection scope of the present application.

Claims

An industrial data processing method (100), used for processing data generated in an industrial production process, the method comprising:

Obtain on-site data generated during industrial production;

performing quality inspection on the on-site data, and obtaining data quality information indicating the quality of the on-site data;

performing data quality improvement processing on the on-site data according to the data quality information, so as to solve quality problems existing in the on-site data;

The on-site data that has undergone data quality improvement processing is stored in a pre-created database.
The method according to claim 1, wherein said performing quality inspection on said on-site data and obtaining data quality information indicating the quality of said on-site data comprises:

calculating a data difference feature of the on-site data, the data difference feature being used to indicate a change trend of the on-site data over time;

Acquire the first field information corresponding to each of the first data quality fields of the field data according to at least one preset first data quality field, and the first data quality fields indicate the fields of the field data quality field;

Acquiring data quality information including the data difference feature and each of the first field information.
The method according to claim 2, wherein, performing data quality improvement processing on the field data according to the data quality information includes:

According to the data difference characteristics included in the data quality information, determine whether the field data has abnormal characteristics, and if the field data has the abnormal characteristics, delete the abnormal data corresponding to the abnormal characteristics of the field data ;

According to the heartbeat signal of the data source of the on-site data, it is judged whether there is data missing in the on-site data, and if there is data missing in the on-site data, data enhancement processing is performed on the on-site data according to a preset data enhancement rule, so as to Supplement missing data in said field data;

According to the first data quality field included in the data quality information, it is judged whether the data format of the field data is the target data format, and if the data format of the field data is not the target data format, the field The data format of the data is converted to the target data format.
The method according to claim 1, said method further comprising:

Obtain the first interaction data output by the first data output terminal in the industrial production system through the presentation layer state conversion interface;

Perform semantic search on the fields in the first interaction data, and judge whether the fields in the first interaction data match the pre-created semantic template;

If the fields in the first interaction data do not match the semantic template, perform data quality improvement processing on the first interaction data, so as to solve the quality problems existing in the first interaction data;

Sending the first interaction data that has undergone data quality improvement processing to the first data receiving end in the industrial production system.
The method according to any one of claims 1 to 4, further comprising:

Obtain the second interaction data output by the second data output terminal in the industrial production system through the message queue;

According to at least one preset second data quality field, respectively acquire second field information corresponding to each of the second data quality fields of the second interaction data, where the second data quality field indicates the first Two fields for the quality of the interaction data;

judging whether there is a data quality problem in the second interaction data according to the information in each second field;

If there is a data quality problem in the second interaction data, perform data quality improvement processing on the second interaction data according to information in each of the second fields, so as to delete abnormal data in the second interaction data. Supplement missing data in the second interactive data, and perform data format conversion on the second interactive data;

Sending the second interaction data that has undergone data quality improvement processing to a second data receiving end in the industrial production system.
An industrial data processing device (600), used for processing data generated in an industrial production process, said device comprising:

An acquisition module (601), configured to acquire on-site data generated during industrial production;

A detection module (602), configured to perform quality detection on the on-site data acquired by the acquisition module (601), and obtain data quality information indicating the quality of the on-site data;

A processing module (603), configured to perform data quality improvement processing on the on-site data according to the data quality information acquired by the detection module (602), so as to solve quality problems existing in the on-site data;

An output module (604), configured to store the on-site data processed by the processing module (603) for data quality improvement into a pre-created database.
An electronic device (700), comprising: a processor (702), a communication interface (704), a memory (706) and a communication bus (708), the processor (702), the memory (706) and the The communication interface (704) completes mutual communication through the communication bus (708);

The memory (706) is used to store at least one executable instruction, and the executable instruction causes the processor (702) to execute the operation corresponding to the industrial data processing method according to any one of claims 1-5.
A computer-readable storage medium, where computer instructions are stored on the computer-readable storage medium, and when the computer instructions are executed by a processor, the processor executes the method described in any one of claims 1-5. method.
A computer program comprising computer-executable instructions which, when executed, cause at least one processor to perform the method according to any one of claims 1-5.
A computer program product tangibly stored on a computer-readable medium and comprising computer-executable instructions which, when executed, cause at least one processor to perform the any one of the methods described.