CN115408465A - Data management system and method based on industrial internet - Google Patents
Data management system and method based on industrial internet Download PDFInfo
- Publication number
- CN115408465A CN115408465A CN202211322233.4A CN202211322233A CN115408465A CN 115408465 A CN115408465 A CN 115408465A CN 202211322233 A CN202211322233 A CN 202211322233A CN 115408465 A CN115408465 A CN 115408465A
- Authority
- CN
- China
- Prior art keywords
- data
- quality inspection
- quality
- industrial internet
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000013523 data management Methods 0.000 title claims abstract description 23
- 238000000034 method Methods 0.000 title claims abstract description 21
- 238000007689 inspection Methods 0.000 claims abstract description 110
- 230000007547 defect Effects 0.000 claims abstract description 11
- 238000007405 data analysis Methods 0.000 claims abstract description 9
- 238000012372 quality testing Methods 0.000 claims abstract description 9
- 238000012423 maintenance Methods 0.000 claims abstract description 4
- 238000013500 data storage Methods 0.000 claims description 7
- 230000005540 biological transmission Effects 0.000 claims description 5
- 230000008521 reorganization Effects 0.000 claims description 5
- 238000012216 screening Methods 0.000 claims description 5
- 230000003287 optical effect Effects 0.000 claims description 4
- 238000005538 encapsulation Methods 0.000 claims description 3
- 230000000717 retained effect Effects 0.000 claims description 3
- 238000003860 storage Methods 0.000 claims description 3
- 238000004458 analytical method Methods 0.000 abstract description 5
- 238000007726 management method Methods 0.000 abstract 1
- 230000002688 persistence Effects 0.000 abstract 1
- 230000002085 persistent effect Effects 0.000 abstract 1
- 238000004519 manufacturing process Methods 0.000 description 4
- 230000002776 aggregation Effects 0.000 description 2
- 238000004220 aggregation Methods 0.000 description 2
- 238000013144 data compression Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24553—Query execution of query operations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/958—Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- General Factory Administration (AREA)
Abstract
The invention provides a data management system and a method based on an industrial internet, comprising the following steps: quality testing machine end, quality testing platform end, clickHouse database and MySQL database, wherein: a plurality of quality inspection machine ends are arranged, and the quality inspection machine ends transmit the quality inspection data detected by the quality inspection machine ends to a quality inspection platform end through an HTTP network; and the quality inspection platform end receives the quality inspection data, deconstructs the quality inspection data and respectively sends the quality inspection data to the Mysql database and the ClickHouse database through the network according to the data types. The problem that comprehensive analysis and detailed data analysis cannot be performed on a large amount of quality inspection data due to the fact that the data of the industrial quality inspection machine is not standard and distributed on respective machines in the field of industrial internet is solved through data persistence and platform, the cost of collection, maintenance, synchronization, statistics and analysis is greatly reduced through data platform access, and automatic and persistent management of quality inspection defect data in the industrial internet scene is achieved.
Description
Technical Field
The invention relates to the field of digital information transmission, in particular to a data management system and method based on an industrial internet.
Background
In the 4.0 industrial era, the informatization technology is not used to improve the efficiency in the links of product production, quality inspection and sale. Because the product quality inspection data structures of the product quality inspection machines and the definitions have differences, the data generated by different machines cannot be summarized and counted, and a platform system is lacked for analyzing the data of different quality inspection machines distributed in different enterprises.
Aiming at the client forms of quality inspection machines distributed in different factories and the problem of different data structures in the product quality inspection link, a complete data structure system is urgently needed, the data structure needs to meet various requirements of a platform on quality inspection data, a unified data structure is formed, the data is uploaded to a platform server through a network, and the platform deconstructs the various data and stores the deconstructed data into a database. And then the platform performs data processing and analysis on the quality inspection data, and finally manages each item of data according to the user requirement and finally displays the data on the platform.
However, the above process has difficulties in that the working environment of the quality inspection machine is often huge product quality inspection, millions of data are often generated in the production process of some part products one day, and the use of the Mysql database is difficult to meet the requirement of reading the huge data by the platform, and in such a data scene, the ClickHouse database is introduced into the platform to maintain the quality inspection data. The ClickHouse database has the following advantages:
1. the data are stored in columns and are suitable for aggregation calculation;
2. the data compression space is large, and IO transmission is reduced;
3. indexes are similar to skip list deconstruction, and the leftmost principle is not required to be met;
4. the writing speed is very fast.
The method has the advantages that the requirement of the platform on the aggregate statistical analysis of various items of quality inspection data is met; when the quality inspection machine uploads data to the platform, the data are written in by a large amount of data Json once, the data compression space is large, the IO transmission efficiency is high, the data writing is rapid, and the query efficiency of a large amount of data is high. However, the ClickHouse definitely does not support things, and has no isolation level, so the final positioning is an analytical database, which just accords with the positioning of a data platform of a quality inspection machine; the ClickHouse does not support high concurrency, and because the platform faces less users, the problem of high concurrency does not exist during reading, which completely accords with the characteristics of the platform. And because the data volume of the quality inspection machine end is huge, the historical data is accumulated too much to cause slow data query after the huge data volume is written, and then the historical database is adopted for data backup, so that the query efficiency is prevented from being gradually reduced due to the excessive historical data.
Disclosure of Invention
Aiming at the defects in the prior art, the invention aims to provide a data management system and method based on the industrial internet.
The invention provides a data management system based on industrial internet, comprising: quality testing machine terminal, quality testing platform terminal, clickHouse database and MySQL database, wherein:
a plurality of quality inspection machine ends are arranged, and the quality inspection machine ends transmit the quality inspection data detected by the quality inspection machine ends to a quality inspection platform end through an HTTP network;
and the quality inspection platform end receives the quality inspection data, deconstructs the quality inspection data and respectively sends the quality inspection data to the Mysql database and the ClickHouse database through the network according to the data types.
Preferably, the quality inspection machine end comprises a data sending module, the quality inspection data is structurally recombined and assembled into a required structure, and the data is sent to the quality inspection platform end through HTTP.
Preferably, the quality inspection platform end comprises a data acquisition module, and is used for screening the received data, removing data with an illegal data structure, and storing the data in a warehouse after screening.
Preferably, the quality inspection platform end analyzes the existing data according to the configuration and the query parameters, performs aggregate query according to different dimensions, and delivers the final data structure to the front end for data display.
The data management method based on the industrial Internet comprises the following steps:
a data structure convention step: the quality inspection platform end and the quality inspection machine end are appointed to acquire a quality inspection data structure;
a data receiving step: the quality inspection platform end receives data packaged and sent by the quality inspection machine end;
a data storage step: analyzing the data, and carrying out structural reorganization, encapsulation and storage on the data to a database;
and (3) data analysis step: analyzing according to quality inspection data retained in the database;
a generation step: and forming a data report by the analyzed data.
Preferably, the data structure stipulating step includes: and the quality inspection machine end encapsulates the quality inspection data according to the structure required by the quality inspection platform end.
Preferably, the data types include quality inspection results, defect data, data of each optical surface, and quality pre-warning.
Preferably, in the data storage step, data storage is performed in a mode of combining Mysql and ClickHouse, and the Mysql stores quality early warning data, product and equipment data; the ClickHouse stores quality inspection result data.
Preferably, in the data analysis step, the quality inspection platform end analyzes the existing data according to the configuration and the query parameters, performs aggregate query according to different dimensions, and sends the final data structure to the front end for data display.
Preferably, the method further comprises the following data backup steps: the timer scans the database, and data beyond the specified time range is transferred to the backup table for historical data maintenance.
Compared with the prior art, the invention has the following beneficial effects:
1. the invention can ensure that the structure and defect definition of data sent at different quality detector ends are kept uniform; and the data from different quality detector ends can be finally counted, analyzed and displayed on the platform according to different dimensions, and the data is provided for a user to be exported according to the data of different dimensions.
2. Compared with the quality inspection machine in the existing single machine mode, the quality inspection data management of the platform is more suitable for the complex situation of the industrial Internet.
3. According to the invention, through data management of an internet scene, the defects that data structures in a single machine mode are different, data statistics in the same dimension is difficult to carry out, and data from different quality inspection machines cannot be analyzed and managed, so that data such as work efficiency, capacity yield and the like of the quality inspection machines of different factories cannot be acquired are overcome.
4. According to the invention, data is persisted through data platformization, and the platformization can greatly improve the supervision and analysis of a factory on a product quality inspection result, so that the production efficiency and the yield are improved.
Drawings
Other features, objects and advantages of the invention will become more apparent upon reading of the detailed description of non-limiting embodiments with reference to the following drawings:
fig. 1 is a schematic diagram of a product quality inspection defect data management system in an industrial internet-oriented scene.
Detailed Description
The present invention will be described in detail with reference to specific examples. The following examples will assist those skilled in the art in further understanding the invention, but are not intended to limit the invention in any way. It should be noted that it would be obvious to those skilled in the art that various changes and modifications can be made without departing from the spirit of the invention. All falling within the scope of the present invention.
As shown in fig. 1, the present invention provides a data management system and method based on industrial internet, the method includes: data structure appointment steps: the quality inspection platform end and the quality inspection machine end are appointed to acquire a quality inspection data structure; a data receiving step: the quality inspection platform end receives data packaged and sent by the quality inspection machine end; a data storage step: analyzing the data, and carrying out structural reorganization, encapsulation and storage on the data to a database; and (3) data analysis step: analyzing according to quality inspection data retained in the database; a generation step: and forming a data report by the analyzed data.
Wherein: in the quality inspection data structure agreed by the system and the quality inspection machine end, all quality inspection machines accessed by the platform need to package the quality inspection result, the defect data, the data of each optical surface, the quality early warning and other data of the machine end according to the structure required by the platform.
The system receives data packaged and sent by a quality detector end, data transmission is carried out through HTTP, the magnitude of the sent data packet is large, and single IO is large.
The data is subjected to structural packaging and stored in a database, the data is stored in a mode of combining Mysql with ClickHouse, the Mysql stores quality early warning data, product and equipment data and the like, and the data is characterized by small data volume, quick access and capability of being modified and maintained at a platform end; and the ClickHouse performs mass-level IO of quality inspection data, and stores the quality inspection result data according to the characteristic of high single-time reading and writing efficiency of the ClickHouse.
The data analysis method based on the Mysql database has the advantages that the data are analyzed according to the data in the database, the data are distributed in the ClickHouse and the Mysql database, data analysis of different dimensions can be conducted on original data according to requirements, data results are flexibly assembled, and the data can be flexibly analyzed according to user requirements.
To explain in detail, the quality testing platform side and the quality testing machine side agree on a set of quality testing data structure: the platform and the quality inspection machine end make a set of data structure, wherein deconstruction of data such as quality inspection result data, defects and details of each optical surface, quality early warning and the like is covered, and the quality inspection machine end is appointed to return data transmitted by HTTP each time according to the deconstruction; the platform receives data transmitted by a quality inspection machine end through HTTP and eliminates illegal data; the platform deconstructs the data and stores the deconstructed data into a Mysql database and a ClickHouse database according to the data types; analyzing the data according to each dimensionality on the platform, inquiring through different aggregation methods, and displaying the final data on the platform; forming a data report by using the analyzed data: the user can export the data as desired.
The invention also provides a product quality inspection defect data management system in an industrial internet-oriented scene, which comprises the following modules:
a data sending end module: the quality inspection machine end carries out structural reorganization and assembly on the quality inspection data to form a required structure, and sends the data to the data receiving module under the platform through HTTP;
a data receiving module: the platform screens the received data, removes the data with an illegal data structure, and puts the data into a warehouse after the screening is finished;
a data analysis module: the platform analyzes the existing data according to the configuration and the query parameters, performs aggregate query according to different dimensions, and hands the final data structure to the front end for data display;
the data backup module: the timer scans the database, and data beyond the specified time range is transferred to the backup table for historical data maintenance, so that the query speed is prevented from slowing due to a large amount of data;
a platform configuration module: data such as shift, equipment, products and the like can be maintained on the platform, and different data can be inquired due to different configured data.
The invention can ensure that the structure and defect definition of data sent at different quality detector ends are kept uniform; and the data from different quality detector ends can be finally counted, analyzed and displayed on the platform according to different dimensions, and the data is provided for a user to be exported according to the data of different dimensions. Compared with the quality inspection machine in the existing single machine mode, the platform quality inspection data management is more suitable for the complex situation of the industrial internet, the data structures in the single machine mode are different, data statistics in the same dimension is difficult to carry out, and the data from different quality inspection machines cannot be analyzed and managed, so that the data of the quality inspection machines in different factories, such as the work efficiency, the productivity, the yield and the like, cannot be acquired; the data platformization is undoubtedly the trend of the industrial internet, the data is duralized, and the platformization can greatly improve the supervision and analysis of a factory on a product quality inspection result, so that the production efficiency and the yield are improved.
Those skilled in the art will appreciate that, in addition to implementing the system and its various devices, modules, units provided by the present invention as pure computer readable program code, the system and its various devices, modules, units provided by the present invention can be fully implemented by logically programming method steps in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers and the like. Therefore, the system and various devices, modules and units thereof provided by the invention can be regarded as a hardware component, and the devices, modules and units included in the system for realizing various functions can also be regarded as structures in the hardware component; means, modules, units for performing the various functions may also be regarded as structures within both software modules and hardware components for performing the method.
The foregoing description of specific embodiments of the present invention has been presented. It is to be understood that the present invention is not limited to the specific embodiments described above, and that various changes or modifications may be made by one skilled in the art within the scope of the appended claims without departing from the spirit of the invention. The embodiments and features of the embodiments of the present application may be combined with each other arbitrarily without conflict.
Claims (10)
1. An industrial internet-based data management system, comprising: quality testing machine end, quality testing platform end, clickHouse database and MySQL database, wherein:
a plurality of quality inspection machine ends are arranged, and the quality inspection machine ends transmit the quality inspection data detected by the quality inspection machine ends to a quality inspection platform end through an HTTP network;
and the quality inspection platform end receives the quality inspection data, deconstructs the quality inspection data and respectively sends the quality inspection data to the Mysql database and the ClickHouse database through the network according to the data types.
2. The industrial internet-based data management system of claim 1, wherein the quality inspection machine side comprises a data transmission module, and the quality inspection machine side performs structural reorganization and assembly on the quality inspection data into a required structure, and transmits the data to the quality inspection platform side through HTTP.
3. The industrial internet-based data management system of claim 2, wherein the quality inspection platform comprises a data acquisition module for screening the received data, removing illegal data of the data structure, and storing the data after the screening is completed.
4. The industrial internet-based data management system of claim 3, wherein the quality inspection platform analyzes existing data according to configuration and query parameters, performs aggregate query according to different dimensions, and hands a final data structure to a front end for data display.
5. A data management method based on industrial Internet is characterized by comprising the following steps:
data structure appointment steps: the quality inspection platform end and the quality inspection machine end are appointed to acquire a quality inspection data structure;
a data receiving step: the quality inspection platform end receives data packaged and sent by the quality inspection machine end;
a data storage step: analyzing the data, and carrying out structural reorganization, encapsulation and storage on the data to a database;
and (3) data analysis step: analyzing according to quality inspection data retained in the database;
a generation step: and forming a data report by the analyzed data.
6. The industrial internet-based data management method of claim 5, wherein the data structure stipulating step comprises: and the quality inspection machine end encapsulates the quality inspection data according to the structure required by the quality inspection platform end.
7. The industrial internet-based data management method according to claim 6, wherein the data types include quality inspection results, defect data, data of each optical surface, quality pre-warning.
8. The industrial internet-based data management method according to claim 5, wherein in the data storage step, data storage is performed in a manner of combining Mysql and ClickHouse, and the Mysql stores quality early warning data, product and equipment data; the ClickHouse stores quality inspection result data.
9. The industrial internet-based data management method according to claim 5, wherein in the data analysis step, the quality inspection platform end analyzes the existing data according to the configuration and the query parameters, performs aggregate query according to different dimensions, and hands the final data structure to the front end for data display.
10. The industrial internet-based data management method according to claim 5, further comprising a data backup step of: the timer scans the database, and data beyond the specified time range is transferred to the backup table for historical data maintenance.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211322233.4A CN115408465A (en) | 2022-10-27 | 2022-10-27 | Data management system and method based on industrial internet |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211322233.4A CN115408465A (en) | 2022-10-27 | 2022-10-27 | Data management system and method based on industrial internet |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115408465A true CN115408465A (en) | 2022-11-29 |
Family
ID=84167624
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211322233.4A Pending CN115408465A (en) | 2022-10-27 | 2022-10-27 | Data management system and method based on industrial internet |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115408465A (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111768850A (en) * | 2020-06-05 | 2020-10-13 | 上海森亿医疗科技有限公司 | Hospital data analysis method, hospital data analysis platform, device and medium |
| CN112462724A (en) * | 2020-12-11 | 2021-03-09 | 四川川能智网实业有限公司 | Data monitoring system based on industrial internet |
| US20220179873A1 (en) * | 2020-12-03 | 2022-06-09 | Boe Technology Group Co., Ltd. | Data management platform, intelligent defect analysis system, intelligent defect analysis method, computer-program product, and method for defect analysis |
-
2022
- 2022-10-27 CN CN202211322233.4A patent/CN115408465A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111768850A (en) * | 2020-06-05 | 2020-10-13 | 上海森亿医疗科技有限公司 | Hospital data analysis method, hospital data analysis platform, device and medium |
| US20220179873A1 (en) * | 2020-12-03 | 2022-06-09 | Boe Technology Group Co., Ltd. | Data management platform, intelligent defect analysis system, intelligent defect analysis method, computer-program product, and method for defect analysis |
| CN112462724A (en) * | 2020-12-11 | 2021-03-09 | 四川川能智网实业有限公司 | Data monitoring system based on industrial internet |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Ferencz et al. | Using Node-RED platform in an industrial environment | |
| CN109726234A (en) | Integrated circuit testing Information Management System based on industry internet | |
| US12066821B2 (en) | Monitoring components of manufacturing application systems with respect to application-specific metrics | |
| CN112016828B (en) | Industrial equipment health management cloud platform architecture based on streaming big data | |
| CN109782714A (en) | Virtual electronic intelligence manufacture plant control system and its control method | |
| CN107508728B (en) | Secondary equipment online monitoring method based on IEC61850 service tracking | |
| CN110908789B (en) | Visual data configuration method and system for multi-source data processing | |
| CN110740079A (en) | full link benchmark test system for distributed scheduling system | |
| CN111367953A (en) | Streaming processing method and device for information data | |
| CN112631754A (en) | Data processing method, data processing device, storage medium and electronic device | |
| Bonnard et al. | A big data/analytics platform for industry 4.0 implementation in SMEs | |
| CN111209314A (en) | System for processing massive log data of power information system in real time | |
| CN115408465A (en) | Data management system and method based on industrial internet | |
| Ljaskovska et al. | Optimization of parameters of technological processes means of the flexsim simulation simulation program | |
| Júnior et al. | Industry 4.0 as a way to enhance Lean Manufacturing and Six Sigma | |
| Iyer et al. | Digitalization: a tool for the successful long-term adoption of lean manufacturing | |
| CN112506887A (en) | Vehicle terminal CAN bus data processing method and device | |
| Comba et al. | Plant Layout and Pick‐and‐place Strategies for Improving Performances in Secondary Packaging Plants of Food Products | |
| CN114387124B (en) | Time sequence data storage method of nuclear power industry internet platform | |
| US10459432B2 (en) | Slicing and merging production events | |
| CN115619169A (en) | Heterogeneous information system integration method and terminal equipment for cross-organization business requirements | |
| CN109976269B (en) | Method for monitoring manufacturing execution process information of rolling and packing workshop | |
| CN105138606A (en) | Server log management method and system | |
| CN114358666A (en) | Quality control method for industrial production process of Sichuan vegetables | |
| Vieira et al. | Are simulation tools ready for big data? Computational experiments with supply chain models developed in simio |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20221129 |
|
| RJ01 | Rejection of invention patent application after publication |