CN117411874A - Industrial data uploading method, device and system, electronic equipment and storage medium - Google Patents
Industrial data uploading method, device and system, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN117411874A CN117411874A CN202311542326.2A CN202311542326A CN117411874A CN 117411874 A CN117411874 A CN 117411874A CN 202311542326 A CN202311542326 A CN 202311542326A CN 117411874 A CN117411874 A CN 117411874A
- Authority
- CN
- China
- Prior art keywords
- data
- industrial data
- industrial
- uploading
- configuration
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 40
- 238000003860 storage Methods 0.000 title claims abstract description 32
- 238000013500 data storage Methods 0.000 claims abstract description 52
- 238000007781 pre-processing Methods 0.000 claims abstract description 8
- 238000004590 computer program Methods 0.000 claims description 7
- 238000006243 chemical reaction Methods 0.000 claims description 4
- 238000004891 communication Methods 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 5
- 238000002474 experimental method Methods 0.000 description 5
- 238000007726 management method Methods 0.000 description 5
- 238000012545 processing Methods 0.000 description 5
- 230000006835 compression Effects 0.000 description 4
- 238000007906 compression Methods 0.000 description 4
- 238000009776 industrial production Methods 0.000 description 4
- 230000004044 response Effects 0.000 description 4
- 238000012550 audit Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 238000000926 separation method Methods 0.000 description 3
- 230000005856 abnormality Effects 0.000 description 2
- 230000003750 conditioning effect Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/12—Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides an industrial data uploading method, device, system, electronic equipment and storage medium, belonging to the technical field of computers, wherein the method comprises the following steps: collecting industrial data; preprocessing industrial data based on preset data storage configuration to obtain preprocessed industrial data, wherein the data storage configuration is used for indicating a file package storage mode of the industrial data; determining an attribute field of industrial data according to preset data uploading configuration, wherein the data uploading configuration is used for indicating the attribute field of the uploaded industrial data; and uploading the data to a cloud server based on the attribute field of the industrial data and the preprocessed industrial data. The industrial data is processed by adopting the data storage configuration and the data uploading configuration, and the uploaded data meets the data configuration requirement, so that the data quality can be improved, the use difficulty of the data can be reduced, and the use efficiency of the industrial data can be improved.
Description
Technical Field
The invention belongs to the technical field of computers, and in particular relates to an industrial data uploading method, an industrial data uploading device, an industrial data uploading system, electronic equipment and a storage medium.
Background
Industrial data refers to various data generated during industrial production, operation, and management. The data may come from aspects of production equipment, sensors, control systems, enterprise management systems, etc., reflecting various aspects and links of industrial production. In the prior art, original industrial data is generally directly uploaded to a cloud, and different original industrial data are difficult to meet unified configuration standards, so that the industrial data stored in the cloud are difficult to use and low in use efficiency.
Disclosure of Invention
Aiming at the problems existing in the prior art, the embodiment of the invention provides an industrial data uploading method, an industrial data uploading device, an industrial data uploading system, electronic equipment and a storage medium.
In a first aspect, the present invention provides an industrial data uploading method, including:
collecting industrial data;
preprocessing the industrial data based on preset data storage configuration to obtain preprocessed industrial data, wherein the data storage configuration is used for indicating a file package storage mode of the industrial data;
determining an attribute field of the industrial data according to preset data uploading configuration, wherein the data uploading configuration is used for indicating the attribute field of the uploaded industrial data;
and uploading the data to a cloud server based on the attribute field of the industrial data and the preprocessed industrial data.
Optionally, the preprocessing the industrial data based on the preset data storage configuration to obtain preprocessed industrial data includes:
determining a target data type and preset data adjustment logic corresponding to the industrial data according to the attribute field of the industrial data;
based on the preset data adjustment logic, adjusting various numerical values in the industrial data to obtain adjusted industrial data;
and carrying out storage mode conversion on the adjusted industrial data based on the data storage configuration corresponding to the target data category, and obtaining the preprocessed industrial data.
Optionally, the data storage configuration includes any one or more of the following sub-items: an upload package format configuration sub-item, an upload package naming configuration sub-item, a single package maximum capacity configuration sub-item, a data introduction file configuration sub-item, a data file directory configuration sub-item, and a file type configuration sub-item;
the uploading file packet format configuration sub-item is used for indicating the storage format of the uploading file packet;
the uploading file package naming configuration sub-item is used for indicating the naming mode of the uploading file package;
the single file package maximum capacity configuration sub-item is used for indicating the maximum capacity of a single uploading file package;
the data introduction file configuration sub-item is used for indicating to generate an introduction file of an uploading file packet and carrying the introduction file in the uploading file packet;
the data file directory configuration subitem is used for indicating a directory structure of the uploading file packet;
the file type configuration sub-item is used for indicating that the uploading file packet contains files in one or more formats.
Optionally, the data upload configuration includes any one or more of the following sub-items: a name configuration sub-item, a summary configuration sub-item, a search keyword configuration sub-item, a sharing party configuration sub-item, an affiliated unit configuration sub-item, a file format configuration sub-item, an uploading time configuration sub-item, a disclosure authority configuration sub-item and a data type configuration sub-item;
the uploading the data to the cloud server based on the attribute field of the industrial data and the preprocessed industrial data comprises the following steps:
fusing and generating target data based on the attribute field of the industrial data and the preprocessed industrial data;
and uploading the target data to the cloud server.
In a second aspect, the present invention also provides an industrial data uploading device, including:
the acquisition module is used for acquiring industrial data;
the pretreatment module is used for carrying out pretreatment on the industrial data based on preset data storage configuration to obtain pretreated industrial data, and the data storage configuration is used for indicating a file package storage mode of the industrial data;
the attribute determining module is used for determining attribute fields of the industrial data according to preset data uploading configuration, wherein the data uploading configuration is used for indicating the attribute fields of the industrial data to be uploaded;
and the uploading module is used for uploading the data to a cloud server based on the attribute field of the industrial data and the preprocessed industrial data.
In a third aspect, the present invention also provides an industrial data storage system comprising: the cloud server and the front-end processor applying any one of the industrial data uploading methods are in communication connection.
Optionally, the cloud server is configured to query second industrial data associated with the first industrial data based on an entity-relationship E-R model and the first industrial data, where the first industrial data and the second industrial data are different types of data;
the entity-relation E-R model is used for representing the association relation between the industrial data stored by the cloud server.
Optionally, the cloud server stores industrial data of a plurality of data categories including case data categories, algorithm data categories, and pure data categories;
the cloud server is further configured to:
receiving target data uploaded by the front-end processor;
determining an association relation corresponding to the target data by analyzing the association between the data based on the attribute fields of the target data and the attribute fields of the industrial data stored by the cloud server;
and updating the entity-relation E-R model based on the association relation corresponding to the target data.
In a fourth aspect, the present invention also provides an electronic device, including: at least one memory for storing a program; at least one processor for executing a memory-stored program, which when executed is adapted to carry out the method described in the first aspect or any one of the possible implementations of the first aspect.
In a fifth aspect, the present invention provides a computer readable storage medium storing a computer program which, when run on a processor, causes the processor to perform the method described in the first aspect or any one of the possible implementations of the first aspect.
It will be appreciated that the advantages of the second to fifth aspects may be found in the relevant description of the first aspect, and are not described here again.
In general, the above technical solutions conceived by the present invention have the following beneficial effects compared with the prior art: by adopting the data storage configuration and the data uploading configuration to process the industrial data, the uploaded data accords with the data configuration requirement, the data quality can be improved, after the data is uploaded to the cloud, the cloud can call the industrial data based on the data configuration requirement, the use difficulty of the data can be reduced, and the use efficiency of the industrial data is improved.
Drawings
In order to more clearly illustrate the invention or the technical solutions of the prior art, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are some embodiments of the invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic flow chart of an industrial data uploading method provided by the invention;
FIG. 2 is a schematic diagram of an industrial data uploading device according to the present invention;
FIG. 3 is a schematic diagram of an industrial data storage system according to the present invention.
Detailed Description
The present invention will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
The terms "first" and "second" and the like in the description and in the claims are used for distinguishing between different objects and not for describing a particular sequential order of objects. For example, the first response message and the second response message, etc. are used to distinguish between different response messages, and are not used to describe a particular order of response messages.
In embodiments of the invention, words such as "exemplary" or "such as" are used to mean serving as an example, instance, or illustration. Any embodiment or design described herein as "exemplary" or "e.g." in an embodiment should not be taken as preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete fashion.
In the description of the embodiments of the present invention, unless otherwise specified, the meaning of "plurality" means two or more, for example, the meaning of a plurality of processing units means two or more, or the like; the plurality of elements means two or more elements and the like.
Next, the technical scheme provided in the embodiment of the present invention is described.
Fig. 1 is a schematic flow chart of an industrial data uploading method provided by the present invention, and as shown in fig. 1, an execution body of the industrial data uploading method may be an electronic device, for example, a front-end processor (the front-end processor may be a gateway device disposed on an edge side) or the like. The method comprises the following steps:
step S101, collecting industrial data.
In particular, raw industrial data, which may be various data generated during industrial production, operation and management, may be collected periodically or aperiodically. The data may come from aspects of production equipment, sensors, control systems, enterprise management systems, etc., reflecting various aspects and links of industrial production.
Step S102, preprocessing industrial data based on preset data storage configuration, and obtaining preprocessed industrial data, wherein the data storage configuration is used for indicating a file package storage mode of the industrial data.
Specifically, one or more data storage configurations (for example, data storage configurations corresponding to each data category) may be preconfigured, the data storage configurations may be used to indicate a storage manner of a file package of the industrial data, after the original industrial data is collected, the original industrial data may be preprocessed based on the data storage configurations, and the preprocessed industrial data may satisfy the storage manner of the file package indicated by the data storage configurations.
Step S103, determining an attribute field of the industrial data according to a preset data uploading configuration, wherein the data uploading configuration is used for indicating the attribute field of the uploaded industrial data.
Specifically, one or more data uploading configurations may be preconfigured (for example, the data uploading configurations corresponding to the data types are preconfigured), the data uploading configurations may be used for indicating attribute fields of uploading industrial data (for example, attribute fields indicating names, abstracts, sharing party information and the like of uploading industrial data), after the original industrial data is collected, corresponding attribute fields of the industrial data may be determined according to the attribute fields to be uploaded indicated by the data uploading configurations (for example, the names, abstracts, sharing party information of the industrial data are determined).
It can be understood that before uploading industrial data to the cloud, the industrial data is processed by adopting data storage configuration and data uploading configuration, so that the computing power of the edge side can be fully utilized, the computing power resource utilization rate of the edge side can be improved, and the computing power bottleneck caused by centralized processing through the cloud server is avoided.
Step S104, uploading the data to the cloud server based on the attribute field of the industrial data and the preprocessed industrial data.
Specifically, after the attribute field of the industrial data and the preprocessed industrial data are obtained, the data to be uploaded can be determined, the data to be uploaded is encrypted and compressed, and then the data value cloud server is uploaded.
The cloud server is a server based on cloud computing technology, and is connected to a data center of a cloud service provider through the internet. The cloud server provides a flexible, scalable and highly available computing resource. The cloud server works on the principle that computing, storage and network resources are virtualized, and the resources can be dynamically allocated and managed according to requirements. The cloud server may be created, configured, and monitored through a management interface or API of the cloud service provider.
Optionally, the encrypted and compressed data is uploaded to the cloud through a message middleware technology, so that the success rate of data transmission can be guaranteed, and an uploading and downloading channel can be scheduled according to the bandwidth and the priority.
Therefore, the industrial data is processed by adopting the data storage configuration and the data uploading configuration, the uploaded data meets the data configuration requirement, the data quality can be improved, after the data is uploaded to the cloud, the cloud can call the industrial data based on the data configuration requirement, the use difficulty of the data can be reduced, and the use efficiency of the industrial data is improved.
Optionally, according to the industrial data uploading method provided by the present invention, based on a preset data storage configuration, industrial data is preprocessed, and preprocessed industrial data is obtained, including:
determining a target data type and preset data adjustment logic corresponding to industrial data according to the attribute field of the industrial data;
based on preset data adjustment logic, adjusting various values in the industrial data to obtain adjusted industrial data;
and carrying out storage mode conversion on the adjusted industrial data based on the data storage configuration corresponding to the target data category, and obtaining the preprocessed industrial data.
Specifically, the attribute field of the industrial data may include a name field, and the target data category corresponding to the industrial data may be determined by classifying and marking the industrial data based on the name field. The keywords in the name may characterize the data category, e.g., the value of the name field includes the keyword "algorithm", and the industrial data may be classified and marked based on the keyword, so as to determine that the data category of the industrial data is the algorithm data category. The data categories of the industrial data may be case data category, algorithm data category, and pure data category (pure data category is data category other than case data category, algorithm data category).
A plurality of preset data adjustment logics can be stored locally in advance, and each preset data adjustment logic has corresponding function description (can be described by keywords and can be described by abstracts); the attribute fields of the industrial data can comprise attribute fields such as abstracts and keywords, and one or more preset data adjustment logics with higher matching degree can be selected from the preset data adjustment logics stored in advance locally by analyzing the matching degree between the attribute fields of the industrial data and the function descriptions corresponding to the preset data adjustment logics, and the preset data adjustment logics corresponding to the industrial data can be used as the preset data adjustment logics corresponding to the industrial data.
The preset data conditioning logic may indicate that if multiple sensors collect temperature, humidity and pressure data at the same point in time, they may be combined into one record (integrating the data for easier analysis and processing).
The preset data adjustment logic may instruct the formation of new, easy-to-apply data sets according to different combinations of characteristic values for the time series data.
The preset data conditioning logic may instruct extraction of numerical items with specific characteristics from the raw industrial data (to reduce the amount of data and increase the processing efficiency).
The preset data adjustment logic may indicate that outliers or invalid values are to be culled.
Correspondingly, based on preset data adjustment logic, various values in the industrial data can be integrated, extracted or removed, and adjusted industrial data can be obtained.
The data types of the industrial data can be case data types, algorithm data types and pure data types, and accordingly, the locally pre-stored data storage configuration can comprise a data storage configuration corresponding to the case data types, a data storage configuration corresponding to the algorithm data types and a data storage configuration corresponding to the pure data types.
For example, the preset data adjustment logic may be determined from data characteristics in the industrial data application scenario by analyzing the industrial data historic usage record.
Optionally, according to the industrial data uploading method provided by the invention, the data storage configuration includes any one or more of the following sub-items: an upload package format configuration sub-item, an upload package naming configuration sub-item, a single package maximum capacity configuration sub-item, a data introduction file configuration sub-item, a data file directory configuration sub-item, and a file type configuration sub-item;
the uploading file packet format configuration sub-item is used for indicating the storage format of the uploading file packet;
the uploading file package naming configuration sub-item is used for indicating the naming mode of the uploading file package;
the single package maximum capacity configuration sub-item is used for indicating the maximum capacity of a single uploading package;
the data introduction file configuration sub-item is used for indicating to generate an introduction file of the uploading file packet and carrying the introduction file in the uploading file packet;
the data file directory configuration subitem is used to indicate a directory structure of the upload file package (e.g., may indicate a directory structure of the data source file in the upload file package);
the file type configuration sub-item is used to indicate that the upload file package contains files in one or more formats.
Illustratively, the data storage configuration includes an upload package format configuration sub-item, an upload package naming configuration sub-item, a single package maximum capacity configuration sub-item, a data introduction file configuration sub-item, a data file directory configuration sub-item, and a file type configuration sub-item. Correspondingly, the storage mode conversion can be performed on the adjusted industrial data based on the uploading file package format configuration subitem, the uploading file package naming configuration subitem, the single file package maximum capacity configuration subitem, the data introduction file configuration subitem, the data file directory configuration subitem and the file type configuration subitem in the data storage configuration corresponding to the target data category, so as to obtain the preprocessed industrial data.
Optionally, according to the industrial data uploading method provided by the invention, the data uploading configuration includes any one or more of the following sub-items: a name configuration sub-item, a summary configuration sub-item, a search keyword configuration sub-item, a sharing party configuration sub-item, an affiliated unit configuration sub-item, a file format configuration sub-item, an uploading time configuration sub-item, a disclosure authority configuration sub-item and a data type configuration sub-item;
uploading data to a cloud server based on the attribute field of the industrial data and the preprocessed industrial data, comprising:
based on the attribute field of the industrial data and the preprocessed industrial data, generating target data in a fusion way;
and uploading the target data to a cloud server.
Illustratively, the data upload configuration includes a name configuration sub-item, a summary configuration sub-item, a search keyword configuration sub-item, a sharer configuration sub-item, an affiliated unit configuration sub-item, a file format configuration sub-item, an upload time configuration sub-item, and a disclosure authority configuration sub-item. Accordingly, the attribute field of the industrial data may be determined based on the name configuration sub-item, the abstract configuration sub-item, the search keyword configuration sub-item, the sharing party configuration sub-item, the belonging unit configuration sub-item, the file format configuration sub-item, the uploading time configuration sub-item, and the disclosure authority configuration sub-item in the data uploading configuration.
Illustratively, the data upload configuration corresponding to the pure data includes 9 fields:
field 1: data name (must fill field), field requirements: within 50 characters, the character types are defined as: english letters, numbers, chinese characters;
field 2: data summary (must fill field), field requirements: within 500 characters, the character type is not limited;
field 3: search keywords (optional field), field requirement: each keyword is within 50 characters in length, comma separation is adopted between keywords, the number of the keywords is at most 5, and the types of the keywords are defined as follows: english letters, numbers, chinese characters;
field 4: a data sharing party (field filling is needed) can automatically associate users and subject groups sharing data;
field 5: units to which data belongs (optional field), field requirements: the character length is less than 50 characters, and the defined character types are as follows: english letters, chinese characters, numbers, brackets, space symbols, dashed lines;
field 6: data file format (mandatory fields), the file format is defined as: txt, xls, xlsx, mat, sql, jpg (or other picture format) may be optional and administrator extensible;
field 7: data upload time (field must be filled);
field 8: whether rights are disclosed (field must be filled);
field 9: data type (optional field), optional type, structured data, unstructured data, optional, administrator extensible.
Under the condition of data uploaded by a cloud server, the data storage configuration corresponding to the pure data comprises the following steps:
uploading file format: a zip file;
maximum capacity of single file: 512MB;
uploading file naming rules: data name_sharer_upload time (_part1). Zip, part content in brackets is only used when one file cannot contain all data, and a Zip tool is required to be used for sub-packet compression, and a Zip file packet contains file type checking: a data introduction file, a data source file directory;
the data introduction file is named: "data usage description", format is: doc, docx;
data source file directory name: the data file and the directory can have other subdirectories; it is checked whether the file within the data file contains a txt, xls, xlsx, mat file.
The data usage description document may be in a unified format, taking experimental data as an example, and includes fields:
(1) Summary of the experiment
The experimental data presented in this section generally includes: a. a subject; b. experiment purpose; c. experimental expectation;
(2) Experimental details
Experimental data are presented in more detail in this section, including: a. experimental system description, experimental system diagram, experimental instrument, experimental configuration, sensor arrangement, etc.; b. the experimental principle is illustrated, which illustrates the method steps adopted in the experiment and the principle used; c. the experimental condition description is used for describing the specific experimental condition of each experimental sub-item, such as working condition, technological parameters and the like; d. the experimental data record records experimental data under various experimental conditions and naming meanings of each piece of data, and the relation between the experimental data record and the naming meanings of each piece of data can be displayed in a chart mode; e. recording the experiment abnormality, recording the abnormality in the experiment and possible reasons.
Under the condition that the front-end processor automatically uploads data, the data storage configuration corresponding to the pure data comprises:
uploading file format: zip file, (front-end client automatically completes compression);
maximum capacity of single file: 512MB (front-end processor client auto-complete logic determination);
uploading file naming rules: zip (automatic naming of the front-end client is completed);
data source file directory structure: data source file/data type/data file name;
naming the data file name: device IP address_device number_data name_acquisition frequency_data length_generation time.
Illustratively, there are 8 fields for the data upload configuration corresponding to the case data category:
field 1: case name (must fill field), field requirements: within 50 characters, the character types are defined as: english letters, numbers, chinese characters;
field 2: case digest (must fill field), field requirements: within 500 characters, the character type is not limited;
field 3: search keywords (optional field), field requirement: each keyword is within 50 characters in length, comma separation is adopted between keywords, the number of the keywords is at most 5, and the types of the keywords are defined as follows: english letters, numbers, chinese characters;
field 4: the case sharing party (field filling is necessary) can automatically associate users and subject groups sharing data;
field 5: unit to which case belongs (optional field), field requirement: the character length is less than 50 characters, and the defined character types are as follows: english letters, chinese characters, numbers, brackets, space symbols, dashed lines;
field 6: case file format (mandatory fields), the system provides the choice, the file format is defined as: ppt, pptx, doc, docx, optionally;
field 7: case upload time (must fill field);
field 8: whether rights are disclosed (field must be filled).
Data storage configuration corresponding to the case data category:
uploading file format: a zip file;
maximum capacity of single file: 32MB;
uploading file naming rules: data name_sharer_upload time (_part1). Zip, part content in brackets is used only when one file cannot contain all data, and a zip tool is required to be used for sub-packet compression;
the Zip package contains a file type check: the format file must contain ppt, pptx, doc, docx, administrator extensible format.
Illustratively, the data upload configuration corresponding to the algorithm (code) data category has 8 fields:
field 1: algorithm name (must fill field), field requirements: within 50 characters, the character types are defined as: english letters, numbers, chinese characters;
field 2: algorithm digest (must fill field), field requirement: within 500 characters, the character type is not limited;
field 3: search keywords (optional field), field requirement: each keyword is within 50 characters in length, comma separation is adopted between keywords, the number of the keywords is at most 5, and the types of the keywords are defined as follows: english letters, numbers, chinese characters;
field 4: the algorithm sharing party (field filling is necessary) can automatically associate users and subject groups sharing data;
field 5: units to which the algorithm belongs (optional field), field requirements: the character length is less than 50 characters, and the defined character types are as follows: english letters, chinese characters, numbers, brackets, space symbols, dashed lines;
field 6: algorithm file format (mandatory field), the file format is defined as: c, py, m, optionally, extensible;
field 7: algorithm upload time (must fill field);
field 8: whether rights are disclosed (field must be filled).
Data storage configuration corresponding to algorithm (code) data category:
uploading file format: a zip file;
maximum capacity of single file: 32MB;
uploading file naming rules: data name_sharer_upload time (_part1). Zip, part content in brackets is used only when one file cannot contain all data, and a zip tool is required to be used for sub-packet compression;
the Zip package contains a file type check: algorithm introduction files and algorithm file directories;
the algorithm introduction file is named: "algorithm description", format: doc, docx;
algorithm source file directory name: algorithm codes, other subdirectories can be arranged under the directory;
checking if the files in the algorithm code directory contain c, py, m files, the administrator may extend the format.
The industrial data uploading device provided by the invention is described below, and the industrial data uploading device described below and the industrial data uploading method described above can be correspondingly referred to each other.
Fig. 2 is a schematic structural diagram of an industrial data uploading device provided by the present invention, as shown in fig. 2, the industrial data uploading device includes an acquisition module 10, a preprocessing module 20, an attribute determining module 30, and an uploading module 40, wherein:
an acquisition module 10 for acquiring industrial data;
the preprocessing module 20 is configured to preprocess the industrial data based on a preset data storage configuration, and obtain preprocessed industrial data, where the data storage configuration is used to indicate a file packet storage manner of the industrial data;
the attribute determining module 30 is configured to determine an attribute field of the industrial data according to a preset data uploading configuration, where the data uploading configuration is used to indicate the uploading of the attribute field of the industrial data;
the uploading module 40 is configured to upload data to the cloud server based on the attribute field of the industrial data and the preprocessed industrial data.
FIG. 3 is a schematic diagram of an industrial data storage system according to the present invention, as shown in FIG. 3, the industrial data storage system includes: the cloud server 60 and the front end processor 50 applying any one of the industrial data uploading methods described above, and the cloud server 60 and the front end processor are in communication connection 50.
Optionally, an audit mechanism can be established to automatically audit the uploaded stored data, algorithms and cases, wherein the main audit data comprises usage specification documents, algorithm usage specification document contents, and whether the data source files and algorithm source files are compliant.
Optionally, according to the industrial data storage system provided by the invention, the cloud server is configured to query second industrial data associated with the first industrial data, based on an Entity-Relationship E-R (Entity-Relationship) model and the first industrial data, where the first industrial data and the second industrial data are different types of data;
the entity-relation E-R model is used for representing the association relation between the industrial data stored by the cloud server.
For example, the first industrial data may belong to a case data category and the second industrial data may belong to an algorithm data category or a pure data category.
For example, the first industrial data may belong to an algorithm data category and the second industrial data may belong to a case data category or a pure data category.
For example, the first industrial data may belong to a pure data category and the second industrial data may belong to a case data category or an algorithm data category.
Optionally, according to the industrial data storage system provided by the invention, the cloud server stores industrial data of multiple data types, wherein the multiple data types include case data types, algorithm data types and pure data types;
the cloud server is further configured to:
receiving target data uploaded by a front-end processor;
determining an association relationship corresponding to the target data by analyzing the association between the data based on the attribute fields of the target data and the attribute fields of the industrial data stored by the cloud server;
and updating the entity-relation E-R model based on the association relation corresponding to the target data.
Alternatively, the cloud server may store the industrial data as personal statistics:
(1) Statistics were performed on an individual basis: case number (number), algorithm number (number), data volume (GB);
(2) Statistics are performed according to the personal time dimension: case number (number), algorithm number (number), data volume (GB);
(3) Statistics is carried out according to the keyword dimension of the individual: case number (number), algorithm number (number), data volume (GB);
(4) Statistics are performed according to the data type dimension of the individual: data volume (GB);
the data type here is structured or unstructured data specified at the time of uploading.
The cloud server may count the stored industrial data by group (one group comprising a plurality of people):
(1) Statistics were performed according to group: case number (number), algorithm number (number), data volume (GB);
(2) Statistics were performed according to the time dimension of the group: case number (number), algorithm number (number), data volume (GB);
(3) Statistics are performed according to the keyword dimension of the group: case number (number), algorithm number (number), data volume (GB);
(4) Statistics are performed according to the data type dimension of the group: data volume (GB).
It should be understood that, the foregoing apparatus is used to perform the method in the foregoing embodiment, and corresponding program modules in the apparatus implement principles and technical effects similar to those described in the foregoing method, and reference may be made to corresponding processes in the foregoing method for the working process of the apparatus, which are not repeated herein.
Based on the method in the above embodiment, the embodiment of the invention provides an electronic device. The apparatus may include: at least one memory for storing programs and at least one processor for executing the programs stored by the memory. Wherein the processor is adapted to perform the method described in the above embodiments when the program stored in the memory is executed.
Based on the method in the above embodiment, the embodiment of the present invention provides a computer-readable storage medium storing a computer program, which when executed on a processor, causes the processor to perform the method in the above embodiment.
Based on the method in the above embodiments, an embodiment of the present invention provides a computer program product, which when run on a processor causes the processor to perform the method in the above embodiments.
It is to be appreciated that the processor in embodiments of the invention may be a central processing unit (centralprocessing unit, CPU), other general purpose processor, digital signal processor (digital signalprocessor, DSP), application specific integrated circuit (application specific integrated circuit, ASIC), field programmable gate array (field programmable gate array, FPGA) or other programmable logic device, transistor logic device, hardware components, or any combination thereof. The general purpose processor may be a microprocessor, but in the alternative, it may be any conventional processor.
The steps of the method in the embodiment of the present invention may be implemented by hardware, or may be implemented by executing software instructions by a processor. The software instructions may be comprised of corresponding software modules that may be stored in random access memory (random access memory, RAM), flash memory, read-only memory (ROM), programmable ROM (PROM), erasable programmable PROM (EPROM), electrically erasable programmable EPROM (EEPROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC.
In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present invention, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in or transmitted across a computer-readable storage medium. The computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by a wired (e.g., coaxial cable, fiber optic, digital Subscriber Line (DSL)), or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a Solid State Disk (SSD)), or the like.
It will be appreciated that the various numerical numbers referred to in the embodiments of the present invention are merely for ease of description and are not intended to limit the scope of the embodiments of the present invention.
It will be readily appreciated by those skilled in the art that the foregoing description is merely a preferred embodiment of the invention and is not intended to limit the invention, but any modifications, equivalents, improvements or alternatives falling within the spirit and principles of the invention are intended to be included within the scope of the invention.
Claims (10)
1. An industrial data uploading method, comprising:
collecting industrial data;
preprocessing the industrial data based on preset data storage configuration to obtain preprocessed industrial data, wherein the data storage configuration is used for indicating a file package storage mode of the industrial data;
determining an attribute field of the industrial data according to preset data uploading configuration, wherein the data uploading configuration is used for indicating the attribute field of the uploaded industrial data;
and uploading the data to a cloud server based on the attribute field of the industrial data and the preprocessed industrial data.
2. The industrial data uploading method according to claim 1, wherein the preprocessing the industrial data based on the preset data storage configuration to obtain preprocessed industrial data comprises:
determining a target data type and preset data adjustment logic corresponding to the industrial data according to the attribute field of the industrial data;
based on the preset data adjustment logic, adjusting various numerical values in the industrial data to obtain adjusted industrial data;
and carrying out storage mode conversion on the adjusted industrial data based on the data storage configuration corresponding to the target data category, and obtaining the preprocessed industrial data.
3. The industrial data uploading method according to claim 2, wherein the data storage configuration comprises any one or more of the following sub-items: an upload package format configuration sub-item, an upload package naming configuration sub-item, a single package maximum capacity configuration sub-item, a data introduction file configuration sub-item, a data file directory configuration sub-item, and a file type configuration sub-item;
the uploading file packet format configuration sub-item is used for indicating the storage format of the uploading file packet;
the uploading file package naming configuration sub-item is used for indicating the naming mode of the uploading file package;
the single file package maximum capacity configuration sub-item is used for indicating the maximum capacity of a single uploading file package;
the data introduction file configuration sub-item is used for indicating to generate an introduction file of an uploading file packet and carrying the introduction file in the uploading file packet;
the data file directory configuration subitem is used for indicating a directory structure of the uploading file packet;
the file type configuration sub-item is used for indicating that the uploading file packet contains files in one or more formats.
4. A method of uploading industrial data according to any of claims 1-3, wherein the data upload configuration comprises any one or more of the following sub-items: a name configuration sub-item, a summary configuration sub-item, a search keyword configuration sub-item, a sharing party configuration sub-item, an affiliated unit configuration sub-item, a file format configuration sub-item, an uploading time configuration sub-item, a disclosure authority configuration sub-item and a data type configuration sub-item;
the uploading the data to the cloud server based on the attribute field of the industrial data and the preprocessed industrial data comprises the following steps:
fusing and generating target data based on the attribute field of the industrial data and the preprocessed industrial data;
and uploading the target data to the cloud server.
5. An industrial data uploading device, comprising:
the acquisition module is used for acquiring industrial data;
the pretreatment module is used for carrying out pretreatment on the industrial data based on preset data storage configuration to obtain pretreated industrial data, and the data storage configuration is used for indicating a file package storage mode of the industrial data;
the attribute determining module is used for determining attribute fields of the industrial data according to preset data uploading configuration, wherein the data uploading configuration is used for indicating the attribute fields of the industrial data to be uploaded;
and the uploading module is used for uploading the data to a cloud server based on the attribute field of the industrial data and the preprocessed industrial data.
6. An electronic device, comprising:
at least one memory for storing a program;
at least one processor for executing the memory-stored program, which processor is adapted to perform the method according to any of claims 1-4 when the memory-stored program is executed.
7. A non-transitory computer readable storage medium storing a computer program, characterized in that the computer program, when run on a processor, causes the processor to perform the method of any of claims 1-4.
8. An industrial data storage system, comprising: the cloud server and the front-end processor applying the industrial data uploading method according to any one of claims 1-4 are in communication connection.
9. The industrial data storage system of claim 8, wherein the cloud server is configured to query a second industrial data associated with the first industrial data based on an entity-relationship E-R model and the first industrial data, the first industrial data and the second industrial data being different categories of data;
the entity-relation E-R model is used for representing the association relation between the industrial data stored by the cloud server.
10. The industrial data storage system of claim 9, wherein the cloud server stores industrial data of a plurality of data categories including case data categories, algorithm data categories, and pure data categories;
the cloud server is further configured to:
receiving target data uploaded by the front-end processor;
determining an association relation corresponding to the target data by analyzing the association between the data based on the attribute fields of the target data and the attribute fields of the industrial data stored by the cloud server;
and updating the entity-relation E-R model based on the association relation corresponding to the target data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311542326.2A CN117411874A (en) | 2023-11-16 | 2023-11-16 | Industrial data uploading method, device and system, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311542326.2A CN117411874A (en) | 2023-11-16 | 2023-11-16 | Industrial data uploading method, device and system, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117411874A true CN117411874A (en) | 2024-01-16 |
Family
ID=89499906
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311542326.2A Pending CN117411874A (en) | 2023-11-16 | 2023-11-16 | Industrial data uploading method, device and system, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117411874A (en) |
-
2023
- 2023-11-16 CN CN202311542326.2A patent/CN117411874A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11582123B2 (en) | Distribution of data packets with non-linear delay | |
| CN106100902B (en) | Cloud index monitoring method and device | |
| US10430111B2 (en) | Optimization for real-time, parallel execution of models for extracting high-value information from data streams | |
| US11775501B2 (en) | Trace and span sampling and analysis for instrumented software | |
| US12008027B2 (en) | Optimization for real-time, parallel execution of models for extracting high-value information from data streams | |
| CN109800207B (en) | Log analysis method, device and equipment and computer readable storage medium | |
| US11188443B2 (en) | Method, apparatus and system for processing log data | |
| CN112559271B (en) | Interface performance monitoring method, device and equipment for distributed application and storage medium | |
| CN110147470B (en) | Cross-machine-room data comparison system and method | |
| CN111400393B (en) | Data processing method and device based on multi-application platform and storage medium | |
| CN112765103B (en) | File analysis method, system, device and equipment | |
| AU2019379163A1 (en) | Approach for a controllable trade-off between cost and availability of indexed data in a cloud log aggregation solution such as Splunk or Sumo | |
| CN112632129A (en) | Code stream data management method, device and storage medium | |
| CN112130996A (en) | Data monitoring control system, method and device, electronic equipment and storage medium | |
| CN113721856A (en) | Digital community management data storage system | |
| CN110543509B (en) | Monitoring system, method and device for user access data and electronic equipment | |
| CN116962516A (en) | Data query method, device, equipment and storage medium | |
| CN117411874A (en) | Industrial data uploading method, device and system, electronic equipment and storage medium | |
| CN107273423B (en) | Multimedia message data processing method, device and system | |
| CN110781232A (en) | Data processing method, data processing device, computer equipment and storage medium | |
| CN116069725A (en) | File migration method, device, apparatus, medium and program product | |
| CN110874349A (en) | File sorting method and device | |
| CN112131198B (en) | Log analysis method and device and electronic equipment | |
| CN110740046B (en) | Method and device for analyzing service contract | |
| US20170180511A1 (en) | Method, system and apparatus for dynamic detection and propagation of data clusters |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |