CN113033492B - Magnetic material preparation process identification and monitoring system based on artificial intelligence - Google Patents
Magnetic material preparation process identification and monitoring system based on artificial intelligence Download PDFInfo
- Publication number
- CN113033492B CN113033492B CN202110456827.3A CN202110456827A CN113033492B CN 113033492 B CN113033492 B CN 113033492B CN 202110456827 A CN202110456827 A CN 202110456827A CN 113033492 B CN113033492 B CN 113033492B
- Authority
- CN
- China
- Prior art keywords
- data
- magnetic material
- material preparation
- furnace
- model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 239000000696 magnetic material Substances 0.000 title claims abstract description 99
- 238000002360 preparation method Methods 0.000 title claims abstract description 87
- 238000012544 monitoring process Methods 0.000 title claims abstract description 77
- 238000013473 artificial intelligence Methods 0.000 title claims abstract description 21
- 238000000034 method Methods 0.000 claims abstract description 195
- 230000008569 process Effects 0.000 claims abstract description 108
- 229910052761 rare earth metal Inorganic materials 0.000 claims abstract description 51
- 150000002910 rare earth metals Chemical class 0.000 claims abstract description 51
- 230000001133 acceleration Effects 0.000 claims abstract description 35
- 238000004458 analytical method Methods 0.000 claims abstract description 17
- 238000012545 processing Methods 0.000 claims abstract description 15
- 238000013528 artificial neural network Methods 0.000 claims abstract description 9
- 238000007781 pre-processing Methods 0.000 claims abstract description 5
- 238000012800 visualization Methods 0.000 claims abstract description 5
- 238000011897 real-time detection Methods 0.000 claims abstract description 4
- 238000001514 detection method Methods 0.000 claims description 72
- 238000004519 manufacturing process Methods 0.000 claims description 70
- 238000003723 Smelting Methods 0.000 claims description 25
- 238000004140 cleaning Methods 0.000 claims description 15
- 238000003062 neural network model Methods 0.000 claims description 15
- 238000007599 discharging Methods 0.000 claims description 14
- RYGMFSIKBFXOCR-UHFFFAOYSA-N Copper Chemical group [Cu] RYGMFSIKBFXOCR-UHFFFAOYSA-N 0.000 claims description 11
- 229910052802 copper Inorganic materials 0.000 claims description 11
- 239000010949 copper Substances 0.000 claims description 11
- 238000004364 calculation method Methods 0.000 claims description 6
- 238000001914 filtration Methods 0.000 claims description 6
- 238000012512 characterization method Methods 0.000 claims description 5
- 238000003066 decision tree Methods 0.000 claims description 5
- 230000006835 compression Effects 0.000 claims description 4
- 238000007906 compression Methods 0.000 claims description 4
- 238000007667 floating Methods 0.000 claims description 3
- 238000009499 grossing Methods 0.000 claims description 3
- 238000005096 rolling process Methods 0.000 claims description 3
- 238000011002 quantification Methods 0.000 claims description 2
- 238000012549 training Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 239000000463 material Substances 0.000 description 5
- 230000006870 function Effects 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 238000011161 development Methods 0.000 description 3
- 238000013139 quantization Methods 0.000 description 3
- 244000007853 Sarothamnus scoparius Species 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 2
- 239000008186 active pharmaceutical agent Substances 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000004134 energy conservation Methods 0.000 description 2
- 230000007613 environmental effect Effects 0.000 description 2
- 238000013213 extrapolation Methods 0.000 description 2
- 239000004744 fabric Substances 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 230000010354 integration Effects 0.000 description 2
- 238000002372 labelling Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000008520 organization Effects 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- HPTJABJPZMULFH-UHFFFAOYSA-N 12-[(Cyclohexylcarbamoyl)amino]dodecanoic acid Chemical compound OC(=O)CCCCCCCCCCCNC(=O)NC1CCCCC1 HPTJABJPZMULFH-UHFFFAOYSA-N 0.000 description 1
- 229910000831 Steel Inorganic materials 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000010009 beating Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000009960 carding Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000007123 defense Effects 0.000 description 1
- 230000003111 delayed effect Effects 0.000 description 1
- 239000000428 dust Substances 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000005206 flow analysis Methods 0.000 description 1
- 238000000227 grinding Methods 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 238000011068 loading method Methods 0.000 description 1
- 238000002844 melting Methods 0.000 description 1
- 230000008018 melting Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000011112 process operation Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 238000005488 sandblasting Methods 0.000 description 1
- 239000002893 slag Substances 0.000 description 1
- 239000010959 steel Substances 0.000 description 1
- 238000003860 storage Methods 0.000 description 1
- 230000001502 supplementing effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- 238000009941 weaving Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/41—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/243—Classification techniques relating to the number of classes
- G06F18/24323—Tree-organised classifiers
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N5/00—Computing arrangements using knowledge-based models
- G06N5/04—Inference or reasoning models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0631—Resource planning, allocation, distributing or scheduling for enterprises or organisations
- G06Q10/06316—Sequencing of tasks or work
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/04—Manufacturing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/46—Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Business, Economics & Management (AREA)
- Human Resources & Organizations (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Strategic Management (AREA)
- Mathematical Physics (AREA)
- Economics (AREA)
- Computing Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Tourism & Hospitality (AREA)
- Entrepreneurship & Innovation (AREA)
- Molecular Biology (AREA)
- Biomedical Technology (AREA)
- Marketing (AREA)
- General Business, Economics & Management (AREA)
- Biophysics (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Game Theory and Decision Science (AREA)
- Educational Administration (AREA)
- Development Economics (AREA)
- Operations Research (AREA)
- Quality & Reliability (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Computational Biology (AREA)
- Manufacturing & Machinery (AREA)
- Primary Health Care (AREA)
- Manufacture Of Iron (AREA)
Abstract
Magnetic material preparation process discernment and monitored control system based on artificial intelligence belongs to discernment and control technical field. The invention aims to solve the problems of low monitoring efficiency and low accuracy rate of the existing magnetic material preparation process which depends on manual judgment. The method comprises a data layer, an algorithm layer and an application layer; the data layer stores data identified and monitored in the preparation process of the magnetic material; the algorithm layer carries out actual magnetic material preparation procedure identification and monitoring inference according to the rare earth magnetic material preparation procedure identification and monitoring AI model; the application layer adopts a real-time video stream analysis frame based on DeepStream, and under the support of a DeepSteam video analysis acceleration library, multi-channel videos are subjected to decoding, preprocessing, batch processing, neural network inference, tracking, visualization, display and stream pushing steps, and real-time detection of processes and real-time monitoring of violation operations are realized based on the algorithm layer. The method is mainly applied to identification and monitoring of the preparation process of the magnetic material.
Description
Technical Field
The invention relates to a magnetic material preparation process identification and monitoring system, and belongs to the technical field of identification and monitoring.
Background
The magnetic material is an important basic functional material, has a wide application range, and has irreplaceable requirements in industries such as electronics, information, electric tools, automobiles, household appliances and the like. Because energy conservation, environmental protection and green development are advocated in the current country, as a clean energy, magnetic materials are more and more widely applied to emerging fields such as energy conservation, environmental protection, new energy, electric automobiles, smart cities and smart earth, and even begin to be applied to military and national defense fields such as robots, unmanned planes, aerospace, satellite remote sensing and the like.
The preparation of magnetic materials is a fundamental link of the magnetic material industry chain and plays a very important role. Although the existing magnetic material production enterprises have mature processes and advanced equipment, the existing magnetic material production enterprises still do not have a set of intelligent magnetic material preparation process identification and monitoring system, so that the existing magnetic material preparation process monitoring is basically realized by means of manual on-site inspection and a human-powered video monitoring system, a large amount of manpower and material resources are needed, the efficiency is low, the conditions that illegal operation cannot be found in time and the like exist, and great potential safety hazards exist. Because the magnetic material preparation process also needs to be monitored, identified and analyzed in the magnetic material preparation process, the existing modes are all manual-based video analysis, and the problems of low efficiency and low accuracy exist.
Disclosure of Invention
The invention aims to solve the problems of low monitoring efficiency and low accuracy rate of the existing magnetic material preparation process which depends on manual judgment.
The magnetic material preparation process identification and monitoring system based on artificial intelligence comprises a data layer, an algorithm layer and an application layer;
the data layer stores data identified and monitored by the preparation process of the magnetic material, and the data set stored in the data layer comprises a production scene target detection data set and a production process characteristic detection data set;
the algorithm layer carries out actual magnetic material preparation procedure identification and monitoring inference according to the rare earth magnetic material preparation procedure identification and monitoring AI model; the rare earth magnetic material preparation process identification and monitoring AI model comprises a rare earth magnetic material preparation production scene target detection model, a smelting furnace production process identification model based on space-time relationship reasoning and a safety detection model;
the method comprises the following steps that a target detection model of a production scene prepared from a rare earth magnetic material detects targets with different scales by adopting a neural network model; the smelting furnace production process procedure identification model based on space-time relationship reasoning utilizes rare earth magnetic materials to prepare a target detection result of a production scene target detection model, and gives a procedure identification result according to the space relationship of main targets in different scenes and the time sequence relationship of targets in different frames; the safety detection model judges whether the safety helmet is worn or not according to the existence of worker characteristic data;
the application layer adopts a real-time video stream analysis frame based on DeepStream, and under the support of a DeepSteam video analysis acceleration library, multi-channel videos are subjected to decoding, preprocessing, batch processing, neural network inference, tracking, visualization, display and stream pushing steps, and real-time detection of processes and real-time monitoring of violation operations are realized based on the algorithm layer.
Furthermore, the application layer is supported by cuda, cudnn and TensorRT acceleration libraries, model compression and model quantitative inference acceleration methods are adopted, the rare earth magnetic material preparation process identification of the algorithm layer and acceleration of the AI model inference stage are realized, and then real-time inference on the video stream is realized.
Further, the process of the application layer for realizing the identification of the rare earth magnetic material preparation process of the algorithm layer and the acceleration of the AI model inference monitoring stage comprises the following steps:
the neural network model in the rare earth magnetic material preparation process identification and monitoring AI model is a file comprising network structure parameters and weight parameters, the parameters are realized through TensorRT, and model acceleration in an inference stage is realized by converting 32 floating point operation into 8-bit integer operation.
Furthermore, the system also comprises an image acquisition subsystem, wherein the image acquisition subsystem comprises a multi-channel video data acquisition unit for acquiring multi-channel video data.
Further, a neural network model adopted by the rare earth magnetic material preparation production scene target detection model is a YOLOv3 real-time target detection network model.
Further, the smelting furnace production process procedure identification model based on the spatio-temporal relation reasoning is realized by adopting a LightGBM decision tree model.
Further, the process of realizing real-time detection of working procedures and real-time monitoring of illegal operation by the application layer based on the algorithm layer comprises the following steps:
1.1, initializing each path of video data;
1.2, reading and storing data information of all identified feature objects in the frame; the characteristic object refers to characteristic data of a target detection data set of an actual production scene, and comprises furnace front characteristics and furnace top characteristics; the characteristic data of the process is the characteristic data detected and stored by a rare earth magnetic material preparation production scene target detection model according to the video data;
1.3, if the frame does not identify any characteristic object, continuously sending the inference result of the previous frame, and then continuously traversing the next frame of data;
1.4, process estimation is carried out according to the characteristic object data of the frame; the process deducing process is realized by using a process identification model of the smelting furnace production process based on spatio-temporal relation reasoning;
1.5, if the frame has worker feature data, judging whether a safety helmet is worn, and executing the step 1.6; if the worker characteristic data does not exist, directly executing the step 1.7;
1.6, carrying out smooth filtering processing on the inferred information of whether the safety helmet is worn, wherein a filtering function Value = alpha flag + (1-alpha) lastValue; wherein Value: a calculated filtered value; alpha: smoothing the filter calculation constant; lastValue: the last calculated filter value; flag: judging the sign information of the unworn safety helmet;
if the Value exceeds 0.5, setting flag information flag of the unworn safety helmet as 1; otherwise, setting flag information flag of the unworn safety helmet as 0;
1.7, judging whether the furnace door is in an opening state, if so, setting a discharge starting mark as 1, and setting a furnace door wiping mark as 0; otherwise, directly executing the step 1.8;
1.8, judging whether the furnace door is in a closing state, and if so, setting a discharge starting mark to be 0; otherwise, directly executing step 1.9;
1.9, judging whether the procedure is a procedure of wiping the furnace door, if so, confirming whether each frame of the furnace door is wiped, and storing related mark information; otherwise, directly executing step 1.10;
1.10, storing and deducing procedure result information and alarm information, and placing the procedure result information and the alarm information into a data information stream;
1.11, calling kafka and sending the stored information to a message queue;
1.12, releasing resources;
1.13, if there is next set of data, return to 1.1.
Further, the process of performing process estimation based on the frame feature object data in step 1.4 includes the following steps:
firstly, judging that the characteristic data of the frame is furnace top data, and executing the following steps of 1.4.1 or 1.4.2:
1.4.1, if the characteristic data of the frame is furnace top data, deducing related procedures of the furnace top, and executing steps 1.4.1.1 or 1.4.1.2:
1.4.1.1, if the crucible dumping characteristic data exists, the crucible dumping procedure is performed;
1.4.1.2, otherwise, either 1.4.1.2.1 or 1.4.1.2.2 are executed:
1.4.1.2.1, if there is fire door characteristic data; execute step 1.4.1.2.1.1 or 1.4.1.2.1.2:
1.4.1.2.1.1, if there is characteristic data of the worker in the furnace mouth, then executing step 1.4.1.2.1.1.1 or 1.4.1.2.1.1.2:
1.4.1.2.1.1.1, if there is characteristic data of the charging basket in the furnace mouth, execute step 1.4.1.2.1.1.1.1 or
1.4.1.2.1.1.1.2：
1.4.1.2.1.1.1.1, if there is worker characteristic data of the hand tool, then the charging process;
1.4.1.2.1.1.1.2, otherwise, cleaning the crucible;
1.4.1.2.1.1.2, otherwise, perform step 1.4.1.2.1.1.2.1 or 1.4.1.2.1.1.2.2:
1.4.1.2.1.1.2.1, if the worker characteristic data of the hand-held tool does not appear, then the crucible cleaning process is carried out;
1.4.1.2.1.1.2.2, otherwise, cleaning the furnace mouth;
1.4.1.2.1.2, otherwise, perform step 1.4.1.2.1.2.1:
1.4.1.2.1.2.1, if the worker characteristic data or the bucket characteristic data exist, the process is a batching process;
1.4.1.2.2, otherwise, the smelting process;
1.4.2, if the characteristic data of the frame is not furnace top data and the characteristic data of the frame is furnace front data, carrying out furnace front related procedure deduction, and executing the step 1.4.2.1 or 1.4.2.2:
1.4.2.1, if the characteristic data of the person in front of the door does not exist, executing the step 1.4.2.1.1 or 1.4.2.1.2:
1.4.2.1.1, if no characterization data exists for the person holding the toolworker, perform step 1.4.2.1.1.1 or 1.4.2.1.1.2:
1.4.2.1.1.1, if there is no characteristic data of the person in front of the furnace, executing step 1.4.2.1.1.1.1 or 1.4.2.1.1.1.2:
1.4.2.1.1.1.1, if there is no oven door outside characteristic data; execute step 1.4.2.1.1.1.1.1 or 1.4.2.1.1.1.1.2:
1.4.2.1.1.1.1.1, if there is no worker feature data, defining an undefined procedure;
1.4.2.1.1.1.1.2, otherwise, perform step 1.4.2.1.1.1.1.2.1 or 1.4.2.1.1.1.1.2.2:
1.4.2.1.1.1.1.2.1, if the furnace characteristic data exist, the procedure is copper rolling;
1.4.2.1.1.1.1.2.2, otherwise, the discharging process is performed;
1.4.2.1.1.1.2, otherwise, perform step 1.4.2.1.1.1.2.1 or 1.4.2.1.1.1.2.2:
1.4.2.1.1.1.2.1, if the safety helmet characteristic data does not exist, the process is a smelting process;
1.4.2.1.1.1.2.2, otherwise, it is a door opening and closing procedure;
1.4.2.1.1.2, otherwise, perform step 1.4.2.1.1.2.1 or 1.4.2.1.1.2.2:
1.4.2.1.1.2.1, if there is no bucket characterizing data, go to step 1.4.2.1.1.2.1.1 or 1.4.2.1.1.2.1.2:
1.4.2.1.1.2.1.1, if there is no characteristic data of the inner side of the oven door, it is the procedure of opening and closing the door;
1.4.2.1.1.2.1.2, otherwise, perform step 1.4.2.1.1.2.1.2.1 or 1.4.2.1.1.2.1.2.2:
1.4.2.1.1.2.1.2.1, if the characteristic data of people in the furnace appear, the procedure is to clean the hearth;
1.4.2.1.1.2.1.2.2, otherwise, the discharging process is performed;
1.4.2.1.1.2.2, otherwise, perform step 1.4.2.1.1.2.2.1 or 1.4.2.1.1.2.2.2:
1.4.2.1.1.2.2.1, if the charging basket and the hearth are separated, cleaning the hearth;
1.4.2.1.1.2.2.2, otherwise, the discharging process is performed;
1.4.2.1.2, otherwise, the discharging process is performed;
1.4.2.2, otherwise, performing steps 1.4.2.2.1, 1.4.2.2.2, or 1.4.2.2.3:
1.4.2.2.1, if there is no copper roll characteristic data, it is a door opening and closing procedure;
1.4.2.2.2, if the copper roll characteristic data exists and no charging basket exists or the charging basket is separated from the hearth, the procedure is to clean the furnace door;
1.4.2.2.3, if the characteristic data of the outside of the furnace door exists and the characteristic data of the hearth does not exist, the procedure is a smelting observation procedure;
1.4.3, returning the process identification result.
Further, the step 1.5 of determining whether to wear the safety helmet includes the following steps:
1.5.1, reading data of a safety helmet characteristic;
1.5.2, reading data of a worker characteristic;
1.5.3, if the worker is not matched, comparing the safety helmet with the position related data of the worker;
1.5.4, if the helmet should belong to the worker, marking that the worker is matched;
1.5.5, if the unread worker data still exists, returning to 1.5.2;
1.5.6, if there is unread helmet data, return to 1.5.1;
1.5.7, traversing all worker data, if there is still a mismatch, then determining that there is an occurrence of an unworn crash helmet condition.
Further, the process of confirming whether each frame of the oven door is wiped in step 1.9 includes the following steps:
1.9.1, reading data information of the oven door characteristics;
1.9.2, reading data information of a worker feature;
1.9.3, comparing the position information of the worker with the position information of four door frames of the oven door, if it is confirmed that a certain door frame is being wiped, storing corresponding data;
1.9.4, if there is any unread worker information, return to 1.9.2;
1.9.5, return the result.
Has the beneficial effects that:
the invention provides a method based on a deep learning network to realize the identification of abnormal processes in the preparation process of a rare earth magnetic material, which not only can realize automatic and informationized identification and monitoring and save the investment of manpower and material resources, but also has more accurate identification and monitoring results, can effectively improve the safety in the production process, reduce the investment of manpower and material resources for safety production of enterprises, reduce the operation cost of the enterprises, and also can avoid negligence caused by human subjective factors so as to reduce the occurrence of safety production accidents; the invention improves the efficiency and the accuracy of identification and monitoring, and also improves the accuracy of monitoring of the magnetic material preparation process, thereby improving the controllability of the magnetic material preparation process and further improving the yield and the yield of the magnetic material preparation.
The method adopts a real-time video stream analysis frame based on DeepStream, adopts inference acceleration methods such as model compression and model quantization and the like to realize acceleration of the neural network model inference stage generated by an algorithm layer, further realizes real-time inference on the video stream, thereby ensuring the real-time property and effectiveness of identification and monitoring, ensuring that process identification and monitoring data are more accurate, and ensuring the identification and monitoring accuracy of a magnetic material preparation process (if data delay exists, the accuracy of the process identification and monitoring can be reduced based on delayed data processing), thereby ensuring the controllability of the magnetic material preparation process, and further helping to improve the yield and the yield of magnetic material preparation.
The method for identifying the computer vision process based on the artificial intelligence can be applied to the preparation of rare earth magnetic materials, and the identification algorithm and the platform provided by the project can be popularized in the related industrial fields.
Drawings
FIG. 1 is a schematic view of a process detection and monitoring system of a rare earth magnetic material preparation smelting furnace based on artificial intelligence;
FIG. 2 is an illustration of a furnace roof;
FIG. 3 is an illustration of a furnace front;
FIG. 4 is a drawing of a picture annotation;
FIG. 5 is a schematic diagram of a network structure of a YOLOv3 real-time target detection model;
FIG. 6 is a diagram of a technical architecture of a rare earth magnetic material preparation process identification and monitoring application system;
FIG. 7 is a schematic diagram of acceleration of the TensorRT implementation inference stage;
FIG. 8 is a schematic diagram of a DeepStream-based intelligent real-time video stream analysis unit;
FIG. 9 is a flow chart of monitoring after identifying a video feature object;
FIG. 10 is a general flowchart of a procedure inference algorithm;
FIG. 11 is a flowchart of a furnace roof process extrapolation algorithm;
FIG. 12 is a flow chart of a stokehole process extrapolation algorithm;
FIG. 13 is a flowchart of a helmet fit inference algorithm;
fig. 14 is a flowchart of an oven door wiping inference algorithm.
Detailed Description
The first specific implementation way is as follows: the present embodiment is described in connection with figure 1,
the embodiment is a magnetic material preparation process identification and monitoring system based on artificial intelligence, which comprises a data layer, an algorithm layer and an application layer;
1. the data layer undertakes the tasks of rare earth magnetic material preparation process identification, monitoring AI model data set acquisition and database manufacturing, and comprises acquisition of a rare earth magnetic material preparation production scene target detection data set, manufacturing of the data set, and a rare earth magnetic material preparation production process identification data set manufacturing process.
The data layer stores data identified and monitored by the preparation process of the magnetic material, and the data set stored in the data layer comprises a production scene target detection data set and a production process characteristic detection data set; the data sets mainly comprise data monitored in actual production and also comprise data used for training identification and monitoring AI models of the rare earth magnetic material preparation process; the embodiment is described in the process of establishing the rare earth magnetic material preparation process identification and monitoring AI model, and the data used for training the rare earth magnetic material preparation process identification and monitoring AI model is described as follows:
a. the production scene target detection data set is used for storing data of production scene target detection, and the production scene target detection data detects key targets of a production scene to establish a basis for process identification. The construction process of the production scene target detection data set comprises the following steps:
data of target detection of a production scene are from monitoring videos and are divided into two video sources, namely a furnace top video source and a furnace top video source, wherein the furnace top video source is shown in figure 2, and the furnace top video source is shown in figure 3;
processing a video of the surveillance video into a jpg format of a picture by using an ffmpeg tool, extracting features in the picture according to the intercepted picture, and defining related names for the features in the picture by a professional to form a feature process name data set; the data set comprises two parts, a stokehole characteristic and a furnace top characteristic, wherein: 20 characteristics are needed to be marked in front of the furnace, namely, a person holding a tool, the inner side of a cover, the outer side of the cover, a hearth, a copper roller, a mask, a tundish, a shovel, a dustpan, a broom, a sand blasting machine, a forklift, a grinding machine, a dust collector, gloves, a hand, a barrel and a lifting hook; the furnace top needs to be marked with 18 characteristics, namely, a person holding a tool, a safety cap, a furnace mouth, a pouring crucible, a barrel, an empty crucible, a charging crucible, a drill, a spade, a dustpan, a cleaning cloth, a pouring opening, gloves, hands, a lifting hook, a mask and a steel broom.
Carrying out frame marking through a visual operation interface by using a data marking tool labelImg, and automatically generating an xml file in a VOC format; when the pictures are labeled, all the characteristic processes in the pictures are selected to be respectively subjected to frame selection labeling, and the specific labeling condition is shown in fig. 4.
Reading all the images and the annotation files, corresponding the images and the annotation files one by one, and then creating two image lists, a training image list train.
b. The production process characteristic detection data set is used for storing the data of the process identification of the smelting furnace in the preparation and production process of the rare earth magnetic material, and is the basis of the process identification. The construction process of the production process characteristic detection data set comprises the following steps:
the production process characteristic detection data come from a smelting furnace production process video, wherein the input data are the content of the production scene target detection data set, including each target and coordinates thereof, and the output is the operation process of the front furnace and the top furnace. Wherein the stokehole process comprises: opening a furnace door, wiping a furnace mouth, wiping the furnace door, removing ash, discharging, beating a copper roller, changing a tundish, closing the furnace door, and carrying out smelting observation; the furnace top procedure comprises the following steps: charging, supplementing a furnace nozzle, removing slag and pouring a crucible.
2. The algorithm layer is the core of the AI model, and the algorithm layer identifies and monitors and deduces the actual magnetic material preparation process according to the rare earth magnetic material preparation process identification and the monitored AI model;
the rare earth magnetic material preparation process identification and monitoring AI model comprises a rare earth magnetic material preparation production scene target detection model, a smelting furnace production process identification model based on space-time relationship reasoning and a safety detection model; the first two models are used for process identification, and AI models (including parameters such as the structure and weight of a neural network) of an algorithm layer establish a basis for artificial intelligence inference of an application layer.
Training a rare earth magnetic material preparation process identification and monitoring AI model by using a data set in a data layer during the process of constructing the rare earth magnetic material preparation process identification and monitoring AI model to obtain model parameters; in the process of identifying and monitoring the actual rare earth magnetic material preparation process, an image acquisition device is used for acquiring images, a data layer provides data for an algorithm layer, the algorithm layer loads an AI model for identifying and monitoring the rare earth magnetic material preparation process, and the data provided by the data layer is used for identifying and monitoring the actual rare earth magnetic material preparation process.
Preparing a production scene target detection model by using a rare earth magnetic material:
the detection of the rare earth magnetic material preparation production scene target has higher requirements on real-time performance, 22 paths of video data can be processed at the same time, and a detection result can be obtained in real time. Based on the production requirement, a YOLOv3 real-time target detection network model is adopted, as shown in fig. 5, a backbone part of the YOLOv3 real-time target detection network model adopts Darknet-53, and 3 feature maps with different scales are adopted in the YOLOv3 for object detection, so that the detection of targets with different scales is realized.
And on the basis of fully training and optimizing the rare earth magnetic material preparation production scene target detection model according to the data set in the data layer, obtaining the structural parameters and model parameters of the network, namely a cfg configuration file and a weights file, respectively, loading the corresponding files when detecting the actual production scene target, and preparing the production scene target detection model by using the rare earth magnetic material to realize detection.
A smelting furnace production process procedure identification model based on space-time relation reasoning:
preparing a target detection result (including the type of a scene main target and image coordinates corresponding to the scene main target) of a production scene target detection model by using a rare earth magnetic material, and giving a procedure identification result according to the spatial relationship (angle and distance) of different scene main targets and the time sequence relationship of different frame targets;
the smelting furnace production process procedure identification model based on the spatio-temporal relation reasoning is realized by adopting a LightGBM decision tree model, and has the following advantages: the method has the advantages of faster training efficiency, low memory use, higher accuracy, support for parallelization learning, capability of processing large-scale data and support for directly using category characteristics.
The method comprises the following steps that a production scene target detection model made of a rare earth magnetic material is used for identifying target characteristic data in a monitoring video; the characteristic data is used as input data of a smelting furnace production process procedure identification model based on spatio-temporal relation reasoning; the two models not only have a sequential association sequence, but also have a matching relation in logic, and the process results deduced by the two models can provide accurate quality monitoring for the production and preparation process of the magnetic material; for example, if the process execution is not standard or the safety measure execution is not standard, if the process execution is not standard, the system can give alarm information, so that the magnetic material preparation process identification method and the magnetic material preparation process monitoring system have good identification effect and monitoring effect.
The safety detection model judges whether the safety helmet is worn or not according to the existence of worker characteristic data;
in fact, in the prior art, identification of a certain component or scene is generally achieved by directly using a neural network model, for example, procedure identification results of different frame targets are achieved by directly using a neural network model, the procedure may be that a corresponding procedure is used as a label in an image marking process, detection is performed based on an image, and a detected feature in the image is a feature possessed by the certain procedure, so that the corresponding procedure can be directly determined, which is more convenient and efficient, and the computational cost can also be reduced; however, the invention finds that various conditions of each process need to be sampled in the training process by using a neural network model for direct identification, and because the invention aims at the identification of the rare earth magnetic material preparation process, in the actual detection process, the field process operation is manually carried out or controlled, so the influence of factors such as manual action and the like, particularly the condition that the process execution is not standard or whether safety measures are not normally executed exists or not, not only is the sampling of all conditions corresponding to each process difficult to ensure, but also the detection result has certain omission, so that the integral detection result and monitoring have problems, and also has greater potential safety hazard. According to the invention, through research and multiple tests, the conditions that the overall detection result and monitoring are problematic due to missed detection can be well solved by adopting the neural network model and the LightGBM decision tree model, not only can the detection result of the neural network model provide good characteristic data for the LightGBM decision tree model, but also various conditions can be effectively classified and processed by the method, so that the accuracy of the detection result is improved; meanwhile, the invention can also monitor the condition that the process execution is not standard or the safety measure execution is not standard, realize that a set of system realizes a plurality of functions, save the cost, equivalently improve the execution efficiency of the whole algorithm, correspondingly save the calculation power, and realize greater benefit by using general hardware equipment.
3. The application layer adopts a real-time video stream analysis frame based on DeepStream, and under the support of a DeepSteam video analysis acceleration library, multi-channel videos form a rare earth magnetic material preparation real-time process identification and monitoring application system through the steps of decoding, preprocessing, batch processing, neural network inference, tracking, visualization, display/stream pushing and the like, so that real-time process detection and real-time illegal operation alarm monitoring are realized. And the application layer adopts inference acceleration methods such as model compression, model quantization and the like under the support of acceleration libraries such as cuda, cudnn, TensorRT and the like to realize acceleration of the neural network model inference stage generated by the algorithm layer, thereby realizing real-time inference on the video stream.
In some embodiments, there are a total of 11 melting furnaces on site, each furnace including 1 stokehole camera, 1 top camera, and a total of 66 video data, depending on the rare earth magnetic material preparation production process requirements.
(1) The identification and monitoring system in the rare earth material preparation process requires that 66 paths of video data can be processed simultaneously in real time, and the single-path frame rate > =1 fps.
(2) And outputting a detection result of the furnace front target in real time, and giving a process identification result.
(3) And outputting a furnace top target detection result in real time, and giving a procedure identification result.
(4) And when abnormal conditions such as the fact that a worker does not wear a safety helmet and the like occur, alarming and capturing pictures.
(5) And pushing a target detection result, a process identification result and an alarm result through the message queue.
(6) And various video outputs are supported, and AI model detection results can be output according to a screen tiling mode, file storage and RTSP plug flow.
The magnetic material preparation process identification and monitoring system based on artificial intelligence of the embodiment needs to be combined with hardware setting to realize a target task, and the hardware selection type is as follows:
the video camera adopts Haokwev video dome camera, H.264/H.265 output and 1920X1080 resolution.
The server adopts a rack server which is provided with 4 pieces of Yingwei Tesla T4 computing cards, double CPUs, a 64G memory and an 8TB hard disk.
The technical architecture of the magnetic material preparation process identification and monitoring system based on artificial intelligence in the embodiment is shown in fig. 6; among them, Plugins (built with open source, 3) rd party NV): plug-ins (built based on open source code, third party NV); DNN: deep Neural Networks (Deep Neural Networks); TensorRT: is an acceleration package made by Yingwei to the own platform; DNN inference/TensorRT plugs: DNN inference technology/TensorRT plug-in; communications plugs: a communication plug-in; video/image capture and processing plugs: a video/image capture and processing plug-in; 3 rd party library plugins: a third party plug-in; analytics-multi-camera, multi-sensor frame: multiple cameras, multiple sensor analysis framework; deepstream in associates, Multi-GPU organization: deepsteam container, multiple GPU orchestration; tracking&analytical across large scale/multi-camera: tracking and analysis across large/multiple cameras; streaming and Batch Analytics: stream and batch analysis; event fabric: event weaving; development Tools: a development tool; end to End reference applications: an end-to-end reference application; app building/configuration tools: an application build/configuration tool; End-End organization duplicates&adaptation guides: end-to-end documentation and modification guidelines; plugin templates, custom IP integration: plug-in templates, self-defining IP integration; deepstream SDK: deepsstream software developmentAn integrated environment; TensorRT: is an acceleration package made by Yingwei to the own platform; multimedia APIs/Video Codec SDK: multimedia API/video codec SDK; imaging&Dewaring library: an imaging and deformation library; metadata&messaging: metadata and messaging; NV associates: an NV container; message bus clients: a message bus client; multi-camera tracking lib: a multi-camera tracking library; linux: a Linux operating system; CUDA: (computer Unified Device Architecture), which is an operating platform proposed by video card vendor NVIDIA; perception infra: a sensing device; jetson, Tesla server (Edge and closed): jetson and Tesla server are products released by video card vendor NVIDIA, including edge computing and cloud products; analytics infra-Edge server, NGC, AWS, Azure: analyze infrastructure-edge servers, NGC, AWS, Azure;
the lowest layer is a hardware layer, and the calculation power of the AI model is provided under the hardware support of the Tesla calculation card. Above this is the linux operating system and the cuda parallel compute acceleration library.
The upper layer is the various components used in real-time video stream analysis. The TensorRT realizes model quantization and acceleration in an inference stage, the codec sdk realizes video coding and decoding acceleration, and the cudnn realizes convolution neural network acceleration and the like. The top layer is an application system to realize service requirements.
Deployment and inference acceleration of AI models: when the offline trained AI model is used in a production environment, deployment and inference acceleration are required, and the deployment and acceleration of the model are realized by adopting the TensrT of nvidia in the embodiment. The neural network model in the AI model is a file comprising network structure parameters and weight parameters, the parameters are realized through TensorRT, and 32 floating point operation is converted into 8-bit integer operation to realize model acceleration in an inference stage.
In order to accelerate the inference of the multi-channel video data, the multi-channel video data is simultaneously sent to an AI model for inference, and then the computation process is accelerated through batch inference, and fig. 7 is a schematic diagram of TensorRT inference acceleration.
When the invention is used for analyzing the video stream, the intelligent real-time video stream analyzing unit based on deep stream is adopted, and the intelligent real-time video stream analyzing unit based on deep stream is developed based on deep stream SDK of nvidia and is realized by adopting C + + language programming. And (4) realizing real-time pushing of the alarm event of the inference result set by adopting a Kafka message queue.
The DeepStream SDK of NVIDIA provides a complete flow analysis toolkit that can be used for AI-based video and image understanding and multi-sensor processing. DeepStream is an integral part of NVIDIA Metropolis, a platform for building end-to-end services and solutions.
Fig. 8 is a diagram of architecture of a smart real-time video stream analysis unit based on deep stream. In the figure, a GPU driver provides a hardware driver, cuda realizes parallel computing acceleration, cudnn realizes convolution neural network acceleration, TensorRT realizes inference quantification and acceleration, and opencv realizes the service requirement of image processing.
Under the support of the above supporting library, the intelligent real-time video stream analysis unit based on deep stream appears as a pipeline with a streaming structure.
The 66 RTSP video data enters the pipeline first, and is decoded in a codec video coding and decoding acceleration library; then preprocessing the video; secondly, processing the multi-channel video in batches through a streamMux plug-in so as to facilitate subsequent batch deduction and use; then, batch inference is realized on the deployed TensorRT neural network model; and then, the business functions of visualization, stream pushing and the like of the result are realized.
And finally, realizing real-time pushing of the inference result and the alarm event by adopting a Kafka message queue.
The actual monitoring process of the magnetic material preparation process identification and monitoring system based on artificial intelligence mainly focuses on the monitoring process after identifying the video characteristic target by the YOLO model and deep stream, as shown in fig. 9, and comprises the following steps:
1.1, initializing each path of video data;
1.2, reading and storing data information of all identified characteristic objects in the frame; the characteristic objects refer to characteristic data of a target detection data set of an actual production scene, and comprise furnace front characteristics (20 types) and furnace top characteristics (18 types); the characteristic data of the actual production scene target detection data set in the process is the characteristic data of the rare earth magnetic material preparation production scene target detection model detected according to the video data and stored in the actual production scene target detection data set in the actual monitoring process;
1.3, if the frame does not identify any characteristic object, continuously sending the inference result of the previous frame, and then continuously traversing the next frame of data;
1.4, process estimation is carried out according to the characteristic object data of the frame; the process deducing process is realized by using a process identification model of the smelting furnace production process based on spatio-temporal relation reasoning;
1.5, if the frame has worker feature data, judging whether a safety helmet is worn, and executing the step 1.6; if the worker characteristic data does not exist, directly executing the step 1.7;
1.6, carrying out smooth filtering processing on the inferred information of whether the safety helmet is worn, wherein a filtering function Value = alpha flag + (1-alpha) lastValue; wherein Value: a calculated filtered value; alpha: smoothing the filter calculation constant; lastValue: the last calculated filtered value; flag: judging the sign information of the unworn safety helmet;
if the Value exceeds 0.5, setting the flag of the unworn safety helmet flag to 1; otherwise, setting flag information flag of the unworn safety helmet as 0;
1.7, judging whether the furnace door is in an opening state, if so, setting a discharge starting mark as 1, and setting a furnace door wiping mark as 0; otherwise, directly executing the step 1.8;
1.8, judging whether the furnace door is in a closing state, and if so, setting a discharge starting mark to be 0; here, marks for recording the start and the end of the whole discharging process are recorded, the identification of a specific process is not carried out, and the identification of a specific detailed process is 1.4; otherwise, directly executing step 1.9;
combining with on-site actual operation tests, arrangement optimization and carding of process monitoring show that the process of opening the furnace door and the process of closing the furnace door are two dynamic processes, and some flag data need to be set after the furnace door is determined to be opened or closed, so that the steps 1.7 to 1.8 are further executed after the process inference is carried out according to the characteristic object data of the frame in the step 1.4.
1.9, judging whether the procedure is a procedure of cleaning the furnace door, if so, determining whether each frame of the furnace door is cleaned, and storing related sign information; otherwise, directly executing step 1.10;
1.10, storing and deducing procedure result information and alarm information, and placing the procedure result information and the alarm information into a data information stream; the result of the inference procedure completed in step 1.4, the alarm information and other flag information are stored, and because the information of the custom type needs to be embedded into the linked list of the deepstream, the information needs to be stored first and then be placed in the linked list;
1.11, calling kafka and sending the stored information to a message queue;
1.12, releasing resources;
1.13, if there is next set of data, return to 1.1.
Further, as shown in fig. 10 to 12, the process of performing process estimation based on the frame feature object data in step 1.4 includes the following steps:
firstly, judging that the characteristic data of the frame is furnace top data, and executing the following steps of 1.4.1 or 1.4.2:
1.4.1, if the characteristic data of the frame is furnace top data, deducing related procedures of the furnace top, and executing steps 1.4.1.1 or 1.4.1.2:
1.4.1.1, if the crucible dumping characteristic data exists, the crucible dumping procedure is performed;
1.4.1.2, otherwise, executing 1.4.1.2.1 or 1.4.1.2.2:
1.4.1.2.1, if there is fire door characteristic data; execute step 1.4.1.2.1.1 or 1.4.1.2.1.2:
1.4.1.2.1.1, if there is characteristic data of the worker in the furnace mouth, then executing step 1.4.1.2.1.1.1 or 1.4.1.2.1.1.2:
1.4.1.2.1.1.1, if there is characteristic data of the charging basket in the furnace mouth, execute step 1.4.1.2.1.1.1.1 or
1.4.1.2.1.1.1.2：
1.4.1.2.1.1.1.1, if there is worker characteristic data of the hand tool, then the charging process;
1.4.1.2.1.1.1.2, otherwise, cleaning the crucible;
1.4.1.2.1.1.2, otherwise, perform step 1.4.1.2.1.1.2.1 or 1.4.1.2.1.1.2.2:
1.4.1.2.1.1.2.1, if the worker characteristic data of the hand-held tool does not appear, then the crucible cleaning process is carried out;
1.4.1.2.1.1.2.2, otherwise, cleaning the furnace mouth;
1.4.1.2.1.2, otherwise, perform step 1.4.1.2.1.2.1:
1.4.1.2.1.2.1, if the worker characteristic data or the bucket characteristic data exist, the process is a batching process;
1.4.1.2.2, otherwise, the smelting process;
1.4.2, if the characteristic data of the frame is not furnace top data and the characteristic data of the frame is furnace front data, carrying out furnace front related procedure deduction, and executing the step 1.4.2.1 or 1.4.2.2:
1.4.2.1, if the characteristic data of the person in front of the door does not exist, executing the step 1.4.2.1.1 or 1.4.2.1.2:
1.4.2.1.1, if no characterization data exists for the person holding the toolworker, perform step 1.4.2.1.1.1 or 1.4.2.1.1.2:
1.4.2.1.1.1, if there is no characteristic data of the person in front of the furnace, executing step 1.4.2.1.1.1.1 or 1.4.2.1.1.1.2:
1.4.2.1.1.1.1, if there is no oven door outside characteristic data; execute step 1.4.2.1.1.1.1.1 or 1.4.2.1.1.1.1.2:
1.4.2.1.1.1.1.1, if there is no worker feature data, defining an undefined procedure;
1.4.2.1.1.1.1.2, otherwise, perform step 1.4.2.1.1.1.1.2.1 or 1.4.2.1.1.1.1.2.2:
1.4.2.1.1.1.1.2.1, if the furnace characteristic data exist, the procedure is copper rolling;
1.4.2.1.1.1.1.2.2, otherwise, the discharging process is performed;
1.4.2.1.1.1.2, otherwise, perform step 1.4.2.1.1.1.2.1 or 1.4.2.1.1.1.2.2:
1.4.2.1.1.1.2.1, if the safety helmet characteristic data does not exist, the process is a smelting process;
1.4.2.1.1.1.2.2, otherwise, it is a door opening and closing procedure;
1.4.2.1.1.2, otherwise, perform step 1.4.2.1.1.2.1 or 1.4.2.1.1.2.2:
1.4.2.1.1.2.1, if there is no bucket characterization data, go to step 1.4.2.1.1.2.1.1 or 1.4.2.1.1.2.1.2:
1.4.2.1.1.2.1.1, if there is no characteristic data of the inner side of the oven door, it is the procedure of opening and closing the door;
1.4.2.1.1.2.1.2, otherwise, perform step 1.4.2.1.1.2.1.2.1 or 1.4.2.1.1.2.1.2.2:
1.4.2.1.1.2.1.2.1, if the characteristic data of people in the furnace appear, the procedure is to clean the hearth;
1.4.2.1.1.2.1.2.2, otherwise, the discharging procedure is carried out;
1.4.2.1.1.2.2, otherwise, perform step 1.4.2.1.1.2.2.1 or 1.4.2.1.1.2.2.2:
1.4.2.1.1.2.2.1, if the charging basket and the hearth are separated, cleaning the hearth;
1.4.2.1.1.2.2.2, otherwise, the discharging process is performed;
1.4.2.1.2, otherwise, the discharging process is performed;
1.4.2.2, otherwise, performing steps 1.4.2.2.1, 1.4.2.2.2, or 1.4.2.2.3:
1.4.2.2.1, if there is no copper roll characteristic data, it is a door opening and closing procedure;
1.4.2.2.2, if the copper roll characteristic data exists and no charging basket exists or the charging basket is separated from the hearth, the procedure is to clean the furnace door;
1.4.2.2.3, if there is characteristic data of the outside of the furnace door and there is no characteristic data of the hearth, it is a smelting observation procedure;
1.4.3, returning the process identification result.
Further, as shown in fig. 13, the step 1.5 of determining whether to wear the safety helmet includes the following steps:
1.5.1, reading data of a safety helmet characteristic;
1.5.2, reading data of a worker characteristic;
1.5.3, if the worker is not matched, comparing the safety helmet with the position related data of the worker;
1.5.4, if the helmet should belong to the worker, marking that the worker is matched;
1.5.5, if the unread worker data still exists, returning to 1.5.2;
1.5.6, if there is unread headgear data, return to 1.5.1;
1.5.7, traversing all worker data, if there is still a mismatch, then determining that there is an occurrence of an unworn crash helmet condition.
Further, as shown in fig. 14, the process of confirming whether each frame of the oven door is rubbed at step 1.9 includes the following steps:
1.9.1, reading data information of the oven door characteristics;
1.9.2, reading data information of a worker feature;
1.9.3, comparing the position information of the worker with the position information of four door frames of the oven door, if confirming that the door frame is erased, storing the corresponding data;
1.9.4, if there is any unread worker information, return to 1.9.2;
1.9.5, return the result.
The present invention is capable of other embodiments and its several details are capable of modifications in various obvious respects, all without departing from the spirit and scope of the present invention.
Claims (10)
1. The magnetic material preparation process identification and monitoring system based on artificial intelligence is characterized by comprising a data layer, an algorithm layer and an application layer;
the data layer stores data identified and monitored by the preparation process of the magnetic material, and the data set stored in the data layer comprises a production scene target detection data set and a production process characteristic detection data set;
the algorithm layer identifies and monitors and deduces the actual magnetic material preparation process according to the rare earth magnetic material preparation process identification and monitoring AI model; the rare earth magnetic material preparation process identification and monitoring AI model comprises a rare earth magnetic material preparation production scene target detection model, a smelting furnace production process identification model based on space-time relationship reasoning and a safety detection model;
the method comprises the following steps that a target detection model of a production scene prepared from a rare earth magnetic material detects targets with different scales by adopting a neural network model; the smelting furnace production process procedure identification model based on space-time relationship reasoning utilizes rare earth magnetic materials to prepare a target detection result of a production scene target detection model, and gives a procedure identification result according to the space relationship of main targets in different scenes and the time sequence relationship of targets in different frames; the safety detection model judges whether the safety helmet is worn or not according to the existence of worker characteristic data;
the application layer adopts a real-time video stream analysis frame based on DeepStream, and under the support of a DeepSteam video analysis acceleration library, multi-channel videos are subjected to decoding, preprocessing, batch processing, neural network inference, tracking, visualization, display and stream pushing steps, and real-time detection of processes and real-time monitoring of violation operations are realized based on the algorithm layer.
2. The magnetic material preparation process identification and monitoring system based on artificial intelligence as claimed in claim 1, wherein the application layer is supported by cuda, cudnn, TensrT acceleration library, and the model compression and model quantification inference acceleration method is adopted to realize acceleration of rare earth magnetic material preparation process identification and monitoring AI model inference stage of algorithm layer, thereby realizing real-time inference of video stream.
3. The artificial intelligence based magnetic material preparation process identification and monitoring system according to claim 2, wherein the process of the application layer for achieving the rare earth magnetic material preparation process identification and monitoring AI model inference phase acceleration for the algorithm layer comprises the following steps:
the neural network model in the rare earth magnetic material preparation process identification and monitoring AI model is a file comprising network structure parameters and weight parameters, the parameters are realized through TensorRT, and model acceleration in an inference stage is realized by converting 32 floating point operation into 8-bit integer operation.
4. The artificial intelligence based magnetic material preparation process identification and monitoring system of claim 1, further comprising an image acquisition subsystem, wherein the image acquisition subsystem comprises a plurality of video data acquisition units for acquiring a plurality of video data.
5. The artificial intelligence based magnetic material preparation process identifying and monitoring system as claimed in claim 1, wherein the neural network model adopted by the rare earth magnetic material preparation production scenario target detection model is a YOLOv3 real-time target detection network model.
6. The artificial intelligence based magnetic material preparation process identification and monitoring system of claim 1, wherein the spatiotemporal relationship inference based smelter production process identification model is implemented using a LightGBM decision tree model.
7. The artificial intelligence based magnetic material preparation process identification and monitoring system according to one of claims 1 to 6, wherein the process of realizing real-time process detection and real-time violation operation monitoring based on the algorithm layer by the application layer comprises the following steps:
1.1, initializing each path of video data;
1.2, reading and storing data information of all identified characteristic objects in the frame; the characteristic object refers to characteristic data of a target detection data set of an actual production scene, and comprises furnace front characteristics and furnace top characteristics; the characteristic data of the process is the characteristic data detected and stored by a rare earth magnetic material preparation production scene target detection model according to the video data;
1.3, if the frame does not identify any characteristic object, continuously sending the inference result of the previous frame, and then continuously traversing the next frame of data;
1.4, process estimation is carried out according to the characteristic object data of the frame; the process deducing process is realized by using a process identification model of the smelting furnace production process based on spatio-temporal relation reasoning;
1.5, if the frame has worker feature data, judging whether a safety helmet is worn, and executing the step 1.6; if the worker characteristic data does not exist, directly executing the step 1.7;
1.6, carrying out smooth filtering processing on the inferred information of whether the safety helmet is worn, wherein a filtering function Value = alpha flag + (1-alpha) lastValue; wherein Value: a calculated filtered value; alpha: smoothing the filter calculation constant; lastValue: the last calculated filter value; flag: judging the sign information of the unworn safety helmet;
if the Value exceeds 0.5, setting the flag of the unworn safety helmet flag to 1; otherwise, setting the flag information flag of the unworn safety helmet as 0;
1.7, judging whether the furnace door is in an opening state, if so, setting a discharge starting mark as 1, and setting a furnace door wiping mark as 0; otherwise, directly executing the step 1.8;
1.8, judging whether the furnace door is in a closing state, and if so, setting a discharge starting mark to be 0; otherwise, directly executing step 1.9;
1.9, judging whether the procedure is a procedure of cleaning the furnace door, if so, determining whether each frame of the furnace door is cleaned, and storing related sign information; otherwise, directly executing step 1.10;
1.10, storing and deducing procedure result information and alarm information, and placing the procedure result information and the alarm information into a data information stream;
1.11, calling kafka and sending the stored information to a message queue;
1.12, releasing resources;
1.13, if there is next set of data, return to 1.1.
8. The system for identifying and monitoring the artificial intelligence based magnetic material preparation process according to claim 7, wherein the process of deducing the process according to the characteristic object data of the current frame in step 1.4 comprises the following steps:
firstly, judging that the characteristic data of the frame is furnace top data, and executing the following steps of 1.4.1 or 1.4.2:
1.4.1, if the characteristic data of the frame is furnace top data, deducing related procedures of the furnace top, and executing steps 1.4.1.1 or 1.4.1.2:
1.4.1.1, if the crucible dumping characteristic data exists, the crucible dumping procedure is performed;
1.4.1.2, otherwise, executing 1.4.1.2.1 or 1.4.1.2.2:
1.4.1.2.1, if there is fire door characteristic data; execute step 1.4.1.2.1.1 or 1.4.1.2.1.2:
1.4.1.2.1.1, if there is characteristic data of the worker in the furnace mouth, then executing step 1.4.1.2.1.1.1 or 1.4.1.2.1.1.2:
1.4.1.2.1.1.1, if there is characteristic data of the charging basket in the furnace mouth, execute step 1.4.1.2.1.1.1.1 or
1.4.1.2.1.1.1.2：
1.4.1.2.1.1.1.1, if there is worker characteristic data of the hand tool, then the charging process;
1.4.1.2.1.1.1.2, otherwise, cleaning the crucible;
1.4.1.2.1.1.2, otherwise, perform step 1.4.1.2.1.1.2.1 or 1.4.1.2.1.1.2.2:
1.4.1.2.1.1.2.1, if the worker characteristic data of the hand-held tool does not appear, then the crucible cleaning process is carried out;
1.4.1.2.1.1.2.2, otherwise, cleaning the furnace mouth;
1.4.1.2.1.2, otherwise, perform step 1.4.1.2.1.2.1:
1.4.1.2.1.2.1, if the characteristic data of the worker or the characteristic data of the charging basket exist, the procedure is a burdening procedure;
1.4.1.2.2, otherwise, the smelting process;
1.4.2, if the characteristic data of the frame is not furnace top data and the characteristic data of the frame is furnace front data, carrying out furnace front related procedure deduction, and executing the step 1.4.2.1 or 1.4.2.2:
1.4.2.1, if the characteristic data of the person in front of the door does not exist, executing the step 1.4.2.1.1 or 1.4.2.1.2:
1.4.2.1.1, if no characterization data exists for the person holding the toolworker, perform step 1.4.2.1.1.1 or 1.4.2.1.1.2:
1.4.2.1.1.1, if there is no characteristic data of the person in front of the furnace, executing step 1.4.2.1.1.1.1 or 1.4.2.1.1.1.2:
1.4.2.1.1.1.1, if there is no oven door outside characteristic data; execute step 1.4.2.1.1.1.1.1 or 1.4.2.1.1.1.1.2:
1.4.2.1.1.1.1.1, if there is no worker feature data, defining an undefined procedure;
1.4.2.1.1.1.1.2, otherwise, perform step 1.4.2.1.1.1.1.2.1 or 1.4.2.1.1.1.1.2.2:
1.4.2.1.1.1.1.2.1, if the furnace characteristic data exist, the procedure is copper rolling;
1.4.2.1.1.1.1.2.2, otherwise, the discharging process is performed;
1.4.2.1.1.1.2, otherwise, perform step 1.4.2.1.1.1.2.1 or 1.4.2.1.1.1.2.2:
1.4.2.1.1.1.2.1, if the safety helmet characteristic data do not exist, the smelting process is carried out;
1.4.2.1.1.1.2.2, otherwise, it is a door opening and closing procedure;
1.4.2.1.1.2, otherwise, perform step 1.4.2.1.1.2.1 or 1.4.2.1.1.2.2:
1.4.2.1.1.2.1, if there is no bucket characterization data, go to step 1.4.2.1.1.2.1.1 or 1.4.2.1.1.2.1.2:
1.4.2.1.1.2.1.1, if there is no characteristic data of the inner side of the oven door, it is the procedure of opening and closing the door;
1.4.2.1.1.2.1.2, otherwise, perform step 1.4.2.1.1.2.1.2.1 or 1.4.2.1.1.2.1.2.2:
1.4.2.1.1.2.1.2.1, if the characteristic data of people in the furnace appear, the procedure is to clean the hearth;
1.4.2.1.1.2.1.2.2, otherwise, the discharging process is performed;
1.4.2.1.1.2.2, otherwise, step 1.4.2.1.1.2.2.1 or 1.4.2.1.1.2.2.2 is performed:
1.4.2.1.1.2.2.1, if the charging basket and the hearth are separated, cleaning the hearth;
1.4.2.1.1.2.2.2, otherwise, the discharging process is performed;
1.4.2.1.2, otherwise, the discharging process is performed;
1.4.2.2, otherwise, performing steps 1.4.2.2.1, 1.4.2.2.2, or 1.4.2.2.3:
1.4.2.2.1, if there is no copper roll characteristic data, it is a door opening and closing procedure;
1.4.2.2.2, if the copper roll characteristic data exists and no charging basket exists or the charging basket is separated from the hearth, the procedure is to clean the furnace door;
1.4.2.2.3, if there is characteristic data of the outside of the furnace door and there is no characteristic data of the hearth, it is a smelting observation procedure;
1.4.3, returning the process identification result.
9. The artificial intelligence based magnetic material preparation process identification and monitoring system according to claim 8, wherein the step 1.5 of determining whether or not to wear a safety helmet comprises the steps of:
1.5.1, reading data of a safety helmet characteristic;
1.5.2, reading data of a worker characteristic;
1.5.3, if the worker is not matched, comparing the safety helmet with the position related data of the worker;
1.5.4, if the helmet should belong to the worker, marking that the worker is matched;
1.5.5, if the unread worker data still exists, returning to 1.5.2;
1.5.6, if there is unread headgear data, return to 1.5.1;
1.5.7, traversing all worker data, if there is still a mismatch, then determining that there is an occurrence of an unworn crash helmet condition.
10. The artificial intelligence based magnetic material preparation process identification and monitoring system according to claim 9, wherein the process of confirming whether each frame of the oven door is rubbed or not in step 1.9 comprises the following steps:
1.9.1, reading data information of the oven door characteristics;
1.9.2, reading data information of a worker feature;
1.9.3, comparing the position information of the worker with the position information of four door frames of the oven door, if confirming that the door frame is erased, storing the corresponding data;
1.9.4, if there is any unread worker information, return to 1.9.2;
1.9.5, return the result.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110456827.3A CN113033492B (en) | 2021-04-25 | 2021-04-25 | Magnetic material preparation process identification and monitoring system based on artificial intelligence |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110456827.3A CN113033492B (en) | 2021-04-25 | 2021-04-25 | Magnetic material preparation process identification and monitoring system based on artificial intelligence |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113033492A CN113033492A (en) | 2021-06-25 |
| CN113033492B true CN113033492B (en) | 2022-08-02 |
Family
ID=76455357
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110456827.3A Active CN113033492B (en) | 2021-04-25 | 2021-04-25 | Magnetic material preparation process identification and monitoring system based on artificial intelligence |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113033492B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114039279A (en) * | 2021-09-29 | 2022-02-11 | 交控科技股份有限公司 | Control cabinet monitoring method and system in rail transit station |
| CN117391549B (en) * | 2023-12-12 | 2024-03-08 | 平利县安得利新材料有限公司 | Method and device for realizing preparation node backtracking of barium sulfate based on data circulation |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108319926A (en) * | 2018-02-12 | 2018-07-24 | 安徽金禾软件股份有限公司 | A kind of the safety cap wearing detecting system and detection method of building-site |
| CN108491759A (en) * | 2018-02-10 | 2018-09-04 | 合肥迪宏自动化有限公司 | A kind of process detection device and its process detection method based on deep learning |
| CN110516636A (en) * | 2019-08-30 | 2019-11-29 | 盈盛智创科技(广州)有限公司 | A kind of monitoring method of process, device, computer equipment and storage medium |
-
2021
- 2021-04-25 CN CN202110456827.3A patent/CN113033492B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108491759A (en) * | 2018-02-10 | 2018-09-04 | 合肥迪宏自动化有限公司 | A kind of process detection device and its process detection method based on deep learning |
| CN108319926A (en) * | 2018-02-12 | 2018-07-24 | 安徽金禾软件股份有限公司 | A kind of the safety cap wearing detecting system and detection method of building-site |
| CN110516636A (en) * | 2019-08-30 | 2019-11-29 | 盈盛智创科技(广州)有限公司 | A kind of monitoring method of process, device, computer equipment and storage medium |
Non-Patent Citations (2)
| Title |
|---|
| 面向精度的分段车间关键工序识别方法及应用;许海生 等;《船舶工程》;20200325;141-146 * |
| 面向车间质量管理的关键工序识别方法;唐任仲 等;《浙江大学学报（工学版）》;20121115;1937-1942 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113033492A (en) | 2021-06-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113033492B (en) | Magnetic material preparation process identification and monitoring system based on artificial intelligence | |
| CN108830837B (en) | Method and device for detecting steel ladle corrosion defect | |
| CN110400387A (en) | A kind of joint method for inspecting, system and storage medium based on substation | |
| CN105100748A (en) | Video monitoring system and video monitoring method | |
| CN112449147B (en) | Video cluster monitoring system of photovoltaic power station and image processing method thereof | |
| CN104966304A (en) | Kalman filtering and nonparametric background model-based multi-target detection tracking method | |
| US10037504B2 (en) | Methods for determining manufacturing waste to optimize productivity and devices thereof | |
| CN112950400A (en) | Data processing platform | |
| JP2008225704A (en) | Work evaluation device, work evaluation method and control program | |
| CN112712519A (en) | Non-contact type machine-made sandstone powder content intelligent real-time detection method and device | |
| CN102413321A (en) | Automatic image-recording system and method | |
| CN111178241A (en) | Intelligent monitoring system and method based on video analysis | |
| CN210666820U (en) | Pedestrian abnormal behavior detection system based on DSP edge calculation | |
| CN112528825A (en) | Station passenger recruitment service method based on image recognition | |
| JP2013143655A (en) | Site condition monitoring system and site image provision method | |
| CN116862843A (en) | Construction site safety risk noninductive inspection method and system | |
| CN115190277A (en) | Safety monitoring method, device and equipment for construction area and storage medium | |
| CN113808107B (en) | Image recommendation method, device, electronic equipment and storage medium | |
| CN109190555B (en) | Intelligent shop patrol system based on picture comparison | |
| CN113850144A (en) | Video reliability automatic inspection system based on image recognition | |
| CN113392770A (en) | Typical violation behavior detection method and system for transformer substation operating personnel | |
| CN112804493A (en) | Intelligent monitoring system for electric power engineering construction | |
| CN105590329B (en) | The Intelligent Measurement algorithm of moving target in a kind of oil gas field video monitoring | |
| CN116665419B (en) | Intelligent fault early warning system and method based on AI analysis in power production operation | |
| CN114332826B (en) | Vehicle image recognition method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |