CN117974051A - Data management method based on industrial big data platform - Google Patents
Data management method based on industrial big data platform Download PDFInfo
- Publication number
- CN117974051A CN117974051A CN202410120189.1A CN202410120189A CN117974051A CN 117974051 A CN117974051 A CN 117974051A CN 202410120189 A CN202410120189 A CN 202410120189A CN 117974051 A CN117974051 A CN 117974051A
- Authority
- CN
- China
- Prior art keywords
- data
- industrial
- maintenance
- information
- tool
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 40
- 238000013523 data management Methods 0.000 title claims abstract description 10
- 238000006243 chemical reaction Methods 0.000 claims abstract description 16
- 238000004140 cleaning Methods 0.000 claims abstract description 11
- 238000009960 carding Methods 0.000 claims abstract description 5
- 238000012423 maintenance Methods 0.000 claims description 102
- 238000012545 processing Methods 0.000 claims description 35
- 230000002159 abnormal effect Effects 0.000 claims description 26
- 238000012790 confirmation Methods 0.000 claims description 13
- 238000007689 inspection Methods 0.000 claims description 12
- 230000005856 abnormality Effects 0.000 claims description 11
- 238000004519 manufacturing process Methods 0.000 claims description 5
- 238000004458 analytical method Methods 0.000 claims description 4
- 238000005516 engineering process Methods 0.000 claims description 4
- 238000012217 deletion Methods 0.000 claims description 3
- 230000037430 deletion Effects 0.000 claims description 3
- 238000007726 management method Methods 0.000 description 9
- 230000000694 effects Effects 0.000 description 5
- 238000004364 calculation method Methods 0.000 description 3
- 230000001788 irregular Effects 0.000 description 3
- 239000000463 material Substances 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 230000003111 delayed effect Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000003860 storage Methods 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- 238000012550 audit Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000007596 consolidation process Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000010485 coping Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013144 data compression Methods 0.000 description 1
- 238000007418 data mining Methods 0.000 description 1
- 238000013499 data model Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 239000002360 explosive Substances 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000009472 formulation Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000012552 review Methods 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
Landscapes
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention belongs to the technical field of industrial Internet of things, and particularly relates to a data management method based on an industrial big data platform, which comprises the following steps: s1, carding metadata information of industrial enterprises; s2, industrial data acquisition is carried out based on the carded metadata information; s3, cleaning and format conversion are carried out on the collected industrial data; s4, carrying out data quality auditing on the data processed in the S3 according to a preset data standard, modifying quality problem data according to a data quality auditing result, and improving data quality to obtain quality industrial data; s5, opening the quality industrial data to the outside. By using the method, the data quality of industrial data can be improved, and the problem of difficult application caused by the data quality is solved.
Description
Technical Field
The invention belongs to the technical field of industrial Internet of things, and particularly relates to a data management method based on an industrial big data platform.
Background
With the rise and application of technologies such as the internet of things, industrial data of industrial enterprises also has come to be increased in an explosive manner. But the rapid growth of industrial data also presents a significant challenge to industrial enterprises.
Industrial data is generated in a plurality of links, so that parts and personnel of enterprises are more, and compared with a conventional enterprise, the industrial data has the problems that the data is messy and difficult to effectively manage and use. Moreover, due to the lack of unified data standards and data management tools, the industrial data also often has the problems of non-uniform standards, low quality, difficult utilization and the like, so that the industrial data does not fully exert the value, and the industrial data has limitations in the application process.
Therefore, how to improve the data quality of industrial data and solve the problem of difficult application caused by the data quality is a current urgent problem to be solved.
Disclosure of Invention
Aiming at the defects of the prior art, the invention provides a data management method based on an industrial large data platform, which can improve the data quality of industrial data and solve the problem of difficult application caused by the data quality.
In order to solve the technical problems, the invention adopts the following technical scheme:
a data management method based on an industrial big data platform comprises the following steps:
s1, carding metadata information of industrial enterprises;
s2, industrial data acquisition is carried out based on the carded metadata information;
s3, cleaning and format conversion are carried out on the collected industrial data;
s4, carrying out data quality auditing on the data processed in the S3 according to a preset data standard, modifying quality problem data according to a data quality auditing result, and improving data quality to obtain quality industrial data;
S5, opening the quality industrial data to the outside.
Compared with the prior art, the invention has the following beneficial effects:
By using the method, firstly, metadata information of a data industrial enterprise is obtained, and then industrial data acquisition is carried out based on the carded metadata information. Therefore, the comprehensive performance of the collected industrial data is ensured, and the classification rationality of the collected industrial data is also ensured. And then, cleaning and format conversion are carried out on the collected industrial data, so that the validity of the processed data can be ensured. And then, carrying out data quality auditing according to a preset data standard, and modifying quality problem data according to a data quality auditing result, so that the data quality can be effectively improved. And finally, the quality industrial data obtained after the body volume is improved is opened to the outside, the service diameter of the quality industrial data is provided, and the effective application of the obtained quality industrial data is ensured.
In summary, the method can improve the data quality of industrial data and solve the problem of difficult application caused by the data quality.
Preferably, in S1, the metadata information includes service metadata information, technology metadata information, and operation metadata information.
Business metadata may describe business implications, business rules, etc. of the data, technical metadata structuring the data, and operational metadata describing operational attributes of the data. By the arrangement, comprehensiveness of data collected later and rationality of classification can be guaranteed, so that needed data can be found accurately and conveniently during subsequent use.
Preferably, in S2, the industrial data includes equipment status data, equipment operation data, equipment log data, production data, and sales data.
In this way, the overall integrity of the acquired data can be ensured.
Preferably, in S2, the collected data includes real-time operation data of the device;
And S31, judging whether the equipment has an abnormality or not by using the real-time operation data of the equipment through a background end and sending maintenance information to a maintenance end when the abnormality exists before S4 after S3.
By the arrangement, the state of the equipment can be monitored through the background end, and maintenance information is timely sent out when abnormality is found, so that the abnormal equipment is timely processed.
Preferably, the maintenance end has a plurality of; the maintenance end is a smart phone loaded with a corresponding APP; the maintenance end is also used for setting a current state, and the current state comprises a working state and an idle state; the maintenance end is also used for sending real-time positioning to the background end; the background end stores the positioning of each device, the positioning among tools, the position information of each unconventional tool in the tools and the image information of each unconventional tool;
preferably, S31 includes:
S311, the background end uses real-time operation data of the equipment to judge whether the equipment is abnormal, if so, the background end uses a preset abnormality analysis model to analyze a corresponding maintenance scheme and a maintenance tool to be used; and judging whether an unconventional tool is needed; go to S312 if no non-conventional tools are needed, and go to S313 if non-conventional tools are needed;
s312, the background end sends maintenance information to a maintenance end which is nearest to the abnormal equipment and in an idle state; the maintenance information comprises the position and maintenance scheme of the abnormal equipment;
s313, the background end is matched with maintenance ends of x idle states closest to the abnormal equipment, and tool information is sent respectively; the content of the tool information includes the required non-conventional tools; after the background terminal sends the tool information for a preset time, judging whether a carrying confirmation signal of the corresponding maintenance terminal is received or not; if received, go to S314, if not received, go to S315;
S314, the background end sends maintenance information to a maintenance end which sends a carrying confirmation signal and is nearest to the abnormal equipment;
S315, the background end sends tool inspection information to a maintenance end in an idle state closest to tools, wherein the content of the tool inspection information comprises required non-conventional tools, required position information of each non-conventional tool and image input prompt information; after the background sends the tool inspection information, if the image sent by the corresponding maintenance end is received, matching the image with the image of the required unconventional tool; and if all the required unconventional tools pass through the matching, sending maintenance information to the maintenance terminal.
Because of the large variety of equipment, different equipment has the same fault type, and maintenance tools required to be carried during overhaul are different; likewise, when the same device fails differently, the maintenance tools that need to be carried are also different. A number of industrial plants require corresponding non-conventional tools for maintenance. Unlike conventional tools, these non-conventional tools are difficult to obtain quickly, except for the tool room of maintenance personnel, due to their corresponding specificity of use and difficulty in replacement. Conventional tools are easier to obtain or find alternatives. If the maintainer arrives at the site but does not carry the corresponding non-conventional tool, the maintainer needs to return to take again or send other maintainers to carry the non-conventional tool again, so that manpower and material resources are consumed, the maintenance time of some equipment can be delayed, the maintenance effect of the equipment is affected, and the production of an industrial enterprise is affected. Therefore, after analyzing the fault type of a specific device, it is necessary to ensure that a maintenance person carries a correct maintenance tool, so that it can be ensured that the maintenance process can be performed accurately. By using the method, the abnormal equipment can be effectively processed in the time as fast as possible under various conditions.
Preferably, in S3, the format conversion includes data conversion by script or data conversion by ETL tool.
Preferably, in S3, the cleaning process includes a repeat data process, a missing data process, and an outlier process;
the repeated data processing comprises the steps of reserving one part of data in the repeated data and deleting the rest of data after the repeated data in the industrial data are arranged;
The missing data processing comprises deleting processing, replacing processing and interpolating processing; the deletion processing comprises deleting the missing observed data when the proportion of the missing observed data in the data of a certain variable of the industrial data is smaller than a preset minimum ratio, and deleting all the data of the variable when the proportion of the missing observed data is larger than a preset maximum ratio; the replacement processing comprises the steps of replacing missing data with a preset constant when the proportion of the missing observed data in certain variable data of the industrial data is larger than or equal to a preset lowest ratio and smaller than or equal to a preset highest ratio; the interpolation processing includes predicting a missing value according to other non-missing variables or observations when the proportion of missing observed data in certain variable data of industrial data is greater than or equal to a preset minimum ratio and less than or equal to a preset maximum ratio;
the outlier handling includes deleting outliers, average replacement, or treating as missing values.
By repeating the data processing, the validity of the processed industrial data on the data content can be ensured after the missing data processing and the abnormal value processing.
Preferably, in S5, the external opening of the quality industrial data is realized by means of API sharing.
By the arrangement, convenience and quickness in using the data after quality improvement can be ensured.
Drawings
For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in further detail with reference to the accompanying drawings, in which:
FIG. 1 is a flow chart of the present method;
fig. 2 is a schematic flow chart in the second embodiment.
Detailed Description
The following is a further detailed description of the embodiments:
Example 1
As shown in fig. 1, the embodiment discloses a data management method based on an industrial big data platform, which comprises the following steps:
S1, carding metadata information of industrial enterprises.
In specific implementation, the metadata information includes service metadata information, technology metadata information and operation metadata information.
The business metadata describes business meaning, business rules, etc. of the data. By specifying the business metadata, people can more easily understand and use the business metadata. Metadata eliminates the ambiguity of data, allows people to have consistent knowledge of the data, avoids 'self-speaking and self-speaking', and further provides support for data analysis and application. Common business metadata are: business definition, business term interpretation, etc.; service index name, calculation caliber, derivative index and the like; rules of a business rule engine, data quality detection rules, a data mining algorithm and the like; security or sensitivity level of data, etc.
Technical metadata is the structuring of data, facilitating the identification, storage, transmission and exchange of data by a computer or database. The technical metadata can serve the developer, so that the developer can more clearly determine the storage and structure of the data, and a foundation is laid for application development and system integration. The technical metadata can also serve service personnel, and the service personnel can find the required data more quickly through the metadata clearing data relationship, so that the source and the destination of the data are analyzed, and the data blood-edge tracing and the influence analysis are supported. Common technical metadata are: physical database table names, column names, field lengths, field types, constraint information, data dependencies, and the like; data storage type, location, data storage file format or data compression type, etc.; the method comprises the steps of field-level blood-edge relation, SQL script information, ETL extraction and loading conversion information, interface programs and the like; scheduling dependencies, progress, and data update frequency, etc.
The operation metadata describes operation attributes of the data, including management departments, management responsibilities, and the like. The clear management attribute is beneficial to the responsibility of data management to departments and individuals, and is the basis of data security management. Common operational metadata are: data owners, users, etc.; data access mode, access time, access limit and the like; data access rights, groups, roles, etc.; results of data processing jobs, system execution logs, and the like; data backup, archiver time, etc.
By the arrangement, comprehensiveness of data collected later and rationality of classification can be guaranteed, so that needed data can be found accurately and conveniently during subsequent use.
S2, industrial data acquisition is carried out based on the carded metadata information.
In particular implementations, the industrial data includes equipment status data, equipment operation data, equipment log data, production data, and sales data. In this way, the overall integrity of the acquired data can be ensured.
S3, cleaning and format conversion are carried out on the collected industrial data.
Wherein the format conversion comprises data conversion through script or data conversion through ETL tool.
The cleaning process includes a repeat data process, a missing data process, and an outlier process.
The repeated data processing comprises the steps of reserving one part of data in the repeated data and deleting the rest of data after the repeated data in the industrial data are arranged.
The missing data processing comprises deleting processing, replacing processing and interpolating processing; the deletion processing comprises deleting the missing observed data when the proportion of the missing observed data in the data of a certain variable of the industrial data is smaller than a preset minimum ratio, and deleting all the data of the variable when the proportion of the missing observed data is larger than a preset maximum ratio; the replacement processing comprises the steps of replacing missing data with a preset constant when the proportion of the missing observed data in certain variable data of the industrial data is larger than or equal to a preset lowest ratio and smaller than or equal to a preset highest ratio; the interpolation processing includes predicting a missing value according to other non-missing variables or observations when the proportion of missing observed data in certain variable data of industrial data is greater than or equal to a preset minimum ratio and less than or equal to a preset maximum ratio.
The outlier handling includes deleting outliers, average replacement, or treating as missing values.
By repeating the data processing, the validity of the processed industrial data on the data content can be ensured after the missing data processing and the abnormal value processing.
In the specific implementation, the data may be cleaned as follows.
Missing value cleaning
Missing values are the most common data problem, and there are many processing methods, generally adopting the following 4 steps:
(1) Determining a range of missing values
And calculating the missing value proportion for each field, and then formulating different strategies according to the missing proportion and the field importance.
(2) Removing unwanted fields
Unwanted fields are deleted directly but backed up. The deleting operation is not directly operated on the original data, but part of the data is extracted to perform model construction, the model effect is checked, and if the effect can be popularized to all the data.
(3) Filling missing value content
This step is the most important one and generally comprises the following ways:
filling with business knowledge or experience, such as the field "i love", you "can be inferred by experience"/filling ";
filling with calculation results of the same field index, such as average number, median, etc.;
the calculation results of the different indexes are used for filling, such as deducing the age through the identification card number, deducing the general location through the postal code of the recipient, etc.
(4) Re-acquiring data
Format and content cleansing
(1) The display formats of time date, numerical value, full half angle and the like are inconsistent.
(2) The content has characters which do not exist; such as letters in the identification number, numbers in the name, etc.
(3) The content is inconsistent with the field content; if the name is written as sex, the identification card number is written as mobile phone number, etc.
Logical error cleaning
(1) Data deduplication
(2) And (5) removing unreasonable values. Such as 200 years of age, or-20 years of age.
(3) Removing unreliable fields; for example, the year and month of birth of the ID card number is 20000101, and the age is 80 years old.
Correlation verification
If there are multiple sources of data, a correlation verification can be performed, which is often used in a multi-data source consolidation process to select accurate feature attributes by verifying the correlation between data. For example, the commodity sales are recorded in a wired manner, and the commodity sales are recorded in a telephone customer service manner, and the commodity sales are associated with the mobile phone number through the name, so that whether the commodity information registered in the same person in a wired manner is consistent with the information investigated by the on-line questionnaire or not is seen.
And S4, carrying out data quality auditing on the data processed in the step S3 according to a preset data standard, modifying quality problem data according to a data quality auditing result, and improving the data quality to obtain quality industrial data.
The formulation process of data normalization generally includes the steps of:
Data standard carding: and carrying out the compiling work of the data standard according to the data requirement, determining the data item, and providing data attribute information by the data standard management execution group according to the required data item.
Data standard review: the data standard management committee examines the data standard manuscript and judges whether the data standard meets the application and management requirements or not and whether the data standard meets the data strategic requirements or not.
And (3) data standard release: after the data standard examination is passed, the data standard management office issues the data standard to the whole company. The process data standard management execution group needs to cooperate to carry out influence evaluation of data standard release on the existing application system and data model, and corresponding coping strategies are made.
Then, data quality audit can be performed: after the data standard is released, a data quality auditing task is constructed according to the data standard through a tool, so that the data quality problem is comprehensively checked, and specific data indexes are positioned.
Improvement of data quality: and modifying the quality problem data according to the data quality auditing result, and improving the data quality.
S5, opening the quality industrial data to the outside.
In the specific implementation, the external opening of the quality industrial data is realized through an API sharing mode. By the arrangement, convenience and quickness in using the data after quality improvement can be ensured.
By using the method, firstly, metadata information of a data industrial enterprise is obtained, and then industrial data acquisition is carried out based on the carded metadata information. Therefore, the comprehensive performance of the collected industrial data is ensured, and the classification rationality of the collected industrial data is also ensured. And then, cleaning and format conversion are carried out on the collected industrial data, so that the validity of the processed data can be ensured. And then, carrying out data quality auditing according to a preset data standard, and modifying quality problem data according to a data quality auditing result, so that the data quality can be effectively improved. And finally, the quality industrial data obtained after the body volume is improved is opened to the outside, the service diameter of the quality industrial data is provided, and the effective application of the obtained quality industrial data is ensured. The method can improve the data quality of industrial data and solve the problem of difficult application caused by the data quality.
Example two
Unlike the first embodiment, in S2 in the present embodiment, the collected data includes real-time operation data of the device;
And S31, judging whether the equipment has an abnormality or not by using the real-time operation data of the equipment through a background end and sending maintenance information to a maintenance end when the abnormality exists before S4 after S3.
In the concrete implementation, the background end is a cloud server; the maintenance end is provided with a plurality of maintenance ends; the maintenance end is a smart phone loaded with a corresponding APP; the maintenance end is also used for setting a current state, and the current state comprises a working state and an idle state; the maintenance end is also used for sending real-time positioning to the background end; the background end stores the positioning of each device, the positioning among tools, the position information of each unconventional tool in the tools and the image information of each unconventional tool;
as shown in fig. 2, S31 includes:
S311, the background end uses real-time operation data of the equipment to judge whether the equipment is abnormal, if so, the background end uses a preset abnormality analysis model to analyze a corresponding maintenance scheme and a maintenance tool to be used; and judging whether an unconventional tool is needed; go to S312 if no non-conventional tools are needed, and go to S313 if non-conventional tools are needed;
s312, the background end sends maintenance information to a maintenance end which is nearest to the abnormal equipment and in an idle state; the maintenance information comprises the position and maintenance scheme of the abnormal equipment;
S313, the background end is matched with maintenance ends of x idle states closest to the abnormal equipment, and tool information is sent respectively; the content of the tool information includes the required non-conventional tools; after the background terminal sends the tool information for a preset time, judging whether a carrying confirmation signal of the corresponding maintenance terminal is received or not; if received, go to S314, if not received, go to S315; in this embodiment, the value of x is 3, and in other embodiments, those skilled in the art may specifically set the value according to specific situations, which will not be described herein.
S314, the background end sends maintenance information to a maintenance end which sends a carrying confirmation signal and is nearest to the abnormal equipment;
S315, the background end sends tool inspection information to a maintenance end in an idle state closest to tools, wherein the content of the tool inspection information comprises required non-conventional tools, required position information of each non-conventional tool and image input prompt information; after the background sends the tool inspection information, if the image sent by the corresponding maintenance end is received, matching the image with the image of the required unconventional tool; and if all the required unconventional tools pass through the matching, sending maintenance information to the maintenance terminal.
Tool principle and effect:
because of the large variety of equipment, different equipment has the same fault type, and maintenance tools required to be carried during overhaul are different; likewise, when the same device fails differently, the maintenance tools that need to be carried are also different. A number of industrial plants require corresponding non-conventional tools for maintenance. Unlike conventional tools, these non-conventional tools are difficult to obtain quickly, except for the tool room of maintenance personnel, due to their corresponding specificity of use and difficulty in replacement. Conventional tools are easier to obtain or find alternatives.
If the maintainer arrives at the site but does not carry the corresponding non-conventional tool, the maintainer needs to return to take again or send other maintainers to carry the non-conventional tool again, so that manpower and material resources are consumed, the maintenance time of some equipment can be delayed, the maintenance effect of the equipment is affected, and the production of an industrial enterprise is affected. Therefore, after analyzing the fault type of a specific device, it is necessary to ensure that a maintenance person carries a correct maintenance tool, so that it can be ensured that the maintenance process can be performed accurately.
In the method, after the background end analyzes that the equipment has abnormality, the background end can analyze the corresponding maintenance scheme and the maintenance tool required to be used. And when no unconventional tool is needed, the maintenance information is sent to the maintenance end which is nearest to the abnormal equipment and in an idle state, so that the abnormal equipment is maintained as soon as possible.
When the irregular tool is needed, the tool information is sent to the maintenance ends in the x idle states nearest to the abnormal equipment. If the carrying confirmation signal (such as text confirmation or virtual button confirmation) of the corresponding maintenance terminal is received after the preset time of sending the tool information, the method is not described herein. If the shoelace confirmation signal is received, the fact that a person in the maintenance end closest to the equipment carries the required irregular tool is indicated, so that maintenance information is sent to the maintenance end which carries the confirmation signal and is closest to the abnormal equipment, and the abnormal equipment can be maintained as soon as possible.
If the background end does not receive the carrying confirmation signal, the background end indicates that maintenance personnel corresponding to the maintenance ends do not meet the condition of going to maintenance. Therefore, the background end sends tool inspection information to a maintenance end in an idle state nearest to the tools, and the tool inspection information comprises the required non-conventional tools and the required position information of each non-conventional tool; maintenance personnel can quickly obtain the required non-conventional tools. Thereafter, to ensure that the maintenance personnel confirms that the correct non-conventional tool is being taken, the tool inspection information also includes image entry prompts (i.e., prompts that prompt the maintenance personnel to upload the required non-conventional tool). After the maintenance personnel obtains the required non-conventional tools, the required non-conventional tools can be photographed and then uploaded to the background end, the background end performs matching confirmation, all the required non-conventional tools pass through the matching, and then maintenance information is sent to the maintenance end. By the mode, maintenance personnel are guaranteed to carry irregular tools which are needed uniformly, and the situation that the tools need to be returned again or other maintenance personnel are dispatched again is prevented, so that manpower and material resources are wasted.
In summary, by using the method, in various cases, it can be ensured that the abnormal equipment is effectively processed in a time as fast as possible.
Finally, it should be noted that the above embodiments are only for illustrating the technical solution of the present invention and not for limiting the technical solution, and those skilled in the art should understand that modifications and equivalents may be made to the technical solution of the present invention without departing from the spirit and scope of the present invention, and all such modifications and equivalents are included in the scope of the claims.
Claims (9)
1. The data management method based on the industrial big data platform is characterized by comprising the following steps of:
s1, carding metadata information of industrial enterprises;
s2, industrial data acquisition is carried out based on the carded metadata information;
s3, cleaning and format conversion are carried out on the collected industrial data;
s4, carrying out data quality auditing on the data processed in the S3 according to a preset data standard, modifying quality problem data according to a data quality auditing result, and improving data quality to obtain quality industrial data;
S5, opening the quality industrial data to the outside.
2. The industrial big data platform based data governance method of claim 1, wherein: in S1, the metadata information includes service metadata information, technology metadata information, and operation metadata information.
3. The industrial big data platform based data governance method of claim 1, wherein: in S2, the industrial data includes equipment status data, equipment operation data, equipment log data, production data, and sales data.
4. A method of industrial big data platform based data governance as claimed in claim 3 and wherein: s2, the acquired data comprise real-time operation data of the equipment;
And S31, judging whether the equipment has an abnormality or not by using the real-time operation data of the equipment through a background end and sending maintenance information to a maintenance end when the abnormality exists before S4 after S3.
5. The industrial big data platform based data governance method of claim 4, wherein: the maintenance end is provided with a plurality of maintenance ends; the maintenance end is a smart phone loaded with a corresponding APP; the maintenance end is also used for setting a current state, and the current state comprises a working state and an idle state; the maintenance end is also used for sending real-time positioning to the background end; the background end stores the positioning of each device, the positioning among tools, the position information of each unconventional tool in the tools and the image information of each unconventional tool.
6. The industrial big data platform based data governance method of claim 5, wherein: s31 includes:
S311, the background end uses real-time operation data of the equipment to judge whether the equipment is abnormal, if so, the background end uses a preset abnormality analysis model to analyze a corresponding maintenance scheme and a maintenance tool to be used; and judging whether an unconventional tool is needed; go to S312 if no non-conventional tools are needed, and go to S313 if non-conventional tools are needed;
s312, the background end sends maintenance information to a maintenance end which is nearest to the abnormal equipment and in an idle state; the maintenance information comprises the position and maintenance scheme of the abnormal equipment;
s313, the background end is matched with maintenance ends of x idle states closest to the abnormal equipment, and tool information is sent respectively; the content of the tool information includes the required non-conventional tools; after the background terminal sends the tool information for a preset time, judging whether a carrying confirmation signal of the corresponding maintenance terminal is received or not; if received, go to S314, if not received, go to S315;
S314, the background end sends maintenance information to a maintenance end which sends a carrying confirmation signal and is nearest to the abnormal equipment;
S315, the background end sends tool inspection information to a maintenance end in an idle state closest to tools, wherein the content of the tool inspection information comprises required non-conventional tools, required position information of each non-conventional tool and image input prompt information; after the background sends the tool inspection information, if the image sent by the corresponding maintenance end is received, matching the image with the image of the required unconventional tool; and if all the required unconventional tools pass through the matching, sending maintenance information to the maintenance terminal.
7. The industrial big data platform based data governance method of claim 1, wherein: in S3, the format conversion includes data conversion by script or data conversion by ETL tool.
8. The industrial big data platform based data governance method of claim 1, wherein: s3, the cleaning processing comprises repeated data processing, missing data processing and abnormal value processing;
the repeated data processing comprises the steps of reserving one part of data in the repeated data and deleting the rest of data after the repeated data in the industrial data are arranged;
The missing data processing comprises deleting processing, replacing processing and interpolating processing; the deletion processing comprises deleting the missing observed data when the proportion of the missing observed data in the data of a certain variable of the industrial data is smaller than a preset minimum ratio, and deleting all the data of the variable when the proportion of the missing observed data is larger than a preset maximum ratio; the replacement processing comprises the steps of replacing missing data with a preset constant when the proportion of the missing observed data in certain variable data of the industrial data is larger than or equal to a preset lowest ratio and smaller than or equal to a preset highest ratio; the interpolation processing includes predicting a missing value according to other non-missing variables or observations when the proportion of missing observed data in certain variable data of industrial data is greater than or equal to a preset minimum ratio and less than or equal to a preset maximum ratio;
the outlier handling includes deleting outliers, average replacement, or treating as missing values.
9. The industrial big data platform based data governance method of claim 1, wherein: in S5, the external opening of the quality industrial data is realized through an API sharing mode.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410120189.1A CN117974051A (en) | 2024-01-29 | 2024-01-29 | Data management method based on industrial big data platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410120189.1A CN117974051A (en) | 2024-01-29 | 2024-01-29 | Data management method based on industrial big data platform |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117974051A true CN117974051A (en) | 2024-05-03 |
Family
ID=90856724
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410120189.1A Pending CN117974051A (en) | 2024-01-29 | 2024-01-29 | Data management method based on industrial big data platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117974051A (en) |
-
2024
- 2024-01-29 CN CN202410120189.1A patent/CN117974051A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109684322B (en) | Data processing system and method for automatic maritime affair auditing | |
| CN112181955A (en) | Data standard management method for information sharing of heavy haul railway comprehensive big data platform | |
| CN112000656A (en) | Intelligent data cleaning method and device based on metadata | |
| US20240095256A1 (en) | Method and system for persisting data | |
| CN115391082A (en) | Abnormal data diagnosis method, system and storage medium | |
| CN114461644A (en) | Data acquisition method and device, electronic equipment and storage medium | |
| CN114880405A (en) | Data lake-based data processing method and system | |
| CN112817958A (en) | Electric power planning data acquisition method and device and intelligent terminal | |
| CN112184003A (en) | Bank counter workload assessment method and device, storage medium and electronic equipment | |
| CN113095647B (en) | Vehicle inspection system | |
| CN112199395A (en) | Artificial intelligence analysis method and system | |
| CN112825165A (en) | Project quality management method and device | |
| CN117974051A (en) | Data management method based on industrial big data platform | |
| CN115525575A (en) | Data automation testing method and system based on Dataworks platform | |
| CN109033196A (en) | A kind of distributed data scheduling system and method | |
| CN111427936B (en) | Report generation method and device, computer equipment and storage medium | |
| CN111178028B (en) | Financial data cleaning method, equipment and storage medium | |
| CN112667469A (en) | Method, system and readable medium for automatically generating diversified big data statistical report | |
| CN112905565A (en) | Database management system and data inspection method | |
| Gill et al. | An open source ETL tool-medium and small scale enterprise ETL (MaSSEETL) | |
| CN113435701B (en) | Method and device for processing consumption quality information | |
| CN117557395B (en) | Research and development cost management and control method, system, electronic equipment and storage medium | |
| CN117332023A (en) | Industrial data management method based on data classification management | |
| CN114066170A (en) | Government data open sharing-oriented problem feedback processing system and method | |
| CN114638534A (en) | Risk processing method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |