CN110884791A - Vision garbage classification system and classification method based on TensorFlow - Google Patents
Vision garbage classification system and classification method based on TensorFlow Download PDFInfo
- Publication number
- CN110884791A CN110884791A CN201911189575.1A CN201911189575A CN110884791A CN 110884791 A CN110884791 A CN 110884791A CN 201911189575 A CN201911189575 A CN 201911189575A CN 110884791 A CN110884791 A CN 110884791A
- Authority
- CN
- China
- Prior art keywords
- garbage
- training
- model
- pictures
- tray
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B65—CONVEYING; PACKING; STORING; HANDLING THIN OR FILAMENTARY MATERIAL
- B65F—GATHERING OR REMOVAL OF DOMESTIC OR LIKE REFUSE
- B65F1/00—Refuse receptacles; Accessories therefor
- B65F1/0033—Refuse receptacles; Accessories therefor specially adapted for segregated refuse collecting, e.g. receptacles with several compartments; Combination of receptacles
- B65F1/0053—Combination of several receptacles
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B65—CONVEYING; PACKING; STORING; HANDLING THIN OR FILAMENTARY MATERIAL
- B65F—GATHERING OR REMOVAL OF DOMESTIC OR LIKE REFUSE
- B65F1/00—Refuse receptacles; Accessories therefor
- B65F1/14—Other constructional features; Accessories
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/217—Validation; Performance evaluation; Active pattern learning techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/40—Software arrangements specially adapted for pattern recognition, e.g. user interfaces or toolboxes therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/10—Image acquisition
- G06V10/12—Details of acquisition arrangements; Constructional details thereof
- G06V10/14—Optical characteristics of the device performing the acquisition or on the illumination arrangements
- G06V10/141—Control of illumination
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/94—Hardware or software architectures specially adapted for image or video understanding
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B65—CONVEYING; PACKING; STORING; HANDLING THIN OR FILAMENTARY MATERIAL
- B65F—GATHERING OR REMOVAL OF DOMESTIC OR LIKE REFUSE
- B65F1/00—Refuse receptacles; Accessories therefor
- B65F1/0033—Refuse receptacles; Accessories therefor specially adapted for segregated refuse collecting, e.g. receptacles with several compartments; Combination of receptacles
- B65F2001/008—Means for automatically selecting the receptacle in which refuse should be placed
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B65—CONVEYING; PACKING; STORING; HANDLING THIN OR FILAMENTARY MATERIAL
- B65F—GATHERING OR REMOVAL OF DOMESTIC OR LIKE REFUSE
- B65F2210/00—Equipment of refuse receptacles
- B65F2210/138—Identification means
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B65—CONVEYING; PACKING; STORING; HANDLING THIN OR FILAMENTARY MATERIAL
- B65F—GATHERING OR REMOVAL OF DOMESTIC OR LIKE REFUSE
- B65F2210/00—Equipment of refuse receptacles
- B65F2210/176—Sorting means
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02W—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO WASTEWATER TREATMENT OR WASTE MANAGEMENT
- Y02W30/00—Technologies for solid waste management
- Y02W30/10—Waste collection, transportation, transfer or storage, e.g. segregated refuse collecting, electric or hybrid propulsion
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Multimedia (AREA)
- Mechanical Engineering (AREA)
- Human Computer Interaction (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a visual garbage classification system and a classification method based on TensorFlow. The system comprises a conveying belt, a first photoelectric switch sensor, a garbage tray, a sliding rail, a camera, a second photoelectric switch sensor, an intelligent garbage can, an NVIDIA machine learning platform and an STM32 controller. The system applies the AI technology to garbage recognition, applies the deep learning visual classification technology to garbage sorting, collects garbage images through the camera, adopts a TensorFlow deep learning framework, and greatly improves the accuracy of garbage recognition through the migration training of a MobileNet SSD model. The STM32 singlechip is handled the identification result, and the control conveyer belt realizes rubbish separation, and the control slide rail has realized the transport and the input of rubbish with the rubbish accuracy in dropping into corresponding intelligent garbage bin, need not artifical the participation, has improved work efficiency and rate of accuracy, has greatly reduced the human cost.
Description
Technical Field
The invention relates to the field of garbage classification, in particular to a visual garbage classification system and a classification method based on TensorFlow.
Background
The garbage disposal mainly adopts three modes of garbage incineration, sanitary landfill and compost treatment. Compared with landfill treatment, the garbage incineration has the advantages of small land occupation, easy site selection, short treatment time, obvious reduction, thorough harmlessness, capability of recovering waste heat of garbage incineration and the like, but the environment is seriously polluted by combustion waste gas and waste residues. Although the sanitary landfill controls the seepage of the garbage and the landfill gas, the sanitary landfill has the defects of wide occupied area and long treatment time. Composting is only applicable to kitchen waste. The three main stream garbage disposal modes do not classify and dispose the recyclable garbage, and great resource waste exists.
The garbage amount in China is large, and most of the garbage is recyclable garbage with recycling value, so that the domestic garbage is recycled, and great benefits are brought to the aspects of economy, society, environment and the like. The garbage classification processing system is a complex system and structurally comprises four links of garbage classification collection, garbage classification transportation, garbage classification processing and garbage classification recycling. The garbage classification treatment needs to be carried out from the source, and the automatic garbage classification and collection is one of effective measures for solving the garbage classification problem. At present, sorting of recyclable garbage mainly depends on manpower, belongs to labor intensive industry, and is low in labor efficiency.
With the continuous development of AI technology, visual recognition technology is applied to various fields. In documents of 'Pengxi, Li Jiale, Li Wan, Liu xing Zhou, Zhang Cheng, Lin Xin and Euragui', garbage recognition and classification research [ J ] Shaoguan college, 2019,40(06) '15-20' in SSD target detection model, garbage classification is carried out, but only garbage classification is realized, garbage sorting and putting are not realized, and practicability is not strong; in addition, the SSD target detection model has large calculation amount and high requirement on hardware, and is not easy to realize on embedded equipment.
Disclosure of Invention
Aiming at the defects of the prior art, the invention aims to provide a visual garbage classification system and a classification method based on TensorFlow.
The technical scheme for solving the technical problem of the system is that the visual garbage classification system based on TensorFlow is provided, and the visual garbage classification system is characterized by comprising a conveyor belt, a first photoelectric switch sensor, a garbage tray, a slide rail, a camera, a second photoelectric switch sensor, an intelligent garbage can, an NVIDIA machine learning platform and an STM32 controller;
the garbage tray is arranged on the sliding rail, so that the garbage tray can move along the sliding rail; one end of the conveyor belt is connected with the garbage throwing port, and the other end of the conveyor belt is connected with the garbage tray; the first photoelectric switch sensor is arranged at the edge of one end, connected with the garbage tray, of the conveyor belt and is used for detecting whether garbage falls to the garbage tray from the conveyor belt or not; the camera is placed in front of the intelligent garbage can through the bracket and used for collecting garbage images; a second photoelectric switch sensor is mounted at the position of the opening of each intelligent garbage can and used for detecting the capacity condition of the intelligent garbage can; the conveyor belt, the first photoelectric switch sensor, the slide rail and the second photoelectric switch sensor are all connected with an STM32 controller; the STM32 controller is connected with the NVIDIA machine learning platform; the camera is connected with the NVIDIA machine learning platform.
The technical scheme for solving the technical problem of the method is to provide a garbage classification method of a visual garbage classification system based on TensorFlow, which is characterized by comprising the following steps:
step one, training an NVIDIA machine learning platform;
(1) making an image recognition training set:
① collecting training pictures, collecting pictures of various garbage objects to obtain training pictures;
② screening training pictures, manually screening the training pictures to obtain a training picture set, wherein the pictures in the training picture set contain clear images of objects, have typical characteristics of the objects and diversify object backgrounds;
③ marking training pictures, marking pictures in the training picture set, identifying garbage in the pictures, and outputting the pictures as XML marking files;
④, generating a data set in a TFrecord format, and generating an XML label file into a uniform TFrecord format file by using a program of a target detection library of TensorFlow to obtain the data set in the TFrecord format;
(2) training is started, and the training process is as follows:
① downloading the pre-training model to the object _ detection folder of the object detection library of Tensflow;
②, modifying the object type file, adding or deleting the object type contained in the corresponding layout map file in the object _ detection/data folder according to the actual object type of the garbage;
③ modifying the model configuration file in object _ detection/samples/configurations, modifying the object type quantity according to the actual training garbage type, specifying the file path of the data set in TFRecord format for training and verification, specifying the storage path of the table map file;
④ adjusting the batch size of the pre-training model according to the configuration of the computer hardware CPU and the memory, starting training, training the pre-training model into a trained model;
⑤ converting the trained model into PB model which can run independently through the model file output program of the object detection library of TensorFlow;
⑥ testing the training result, testing the PB model by a testing program of a target detection library of TensorFlow, and increasing the number of the labeled pictures of each object and adjusting the parameter values in the model configuration file to ensure that the recognition rate of the PB model to the pictures is increased to a preset value;
(3) the specific steps of the running environment configuration of the NVIDIA machine learning platform are as follows:
①, installing a neural network reasoning acceleration framework TensorRT of NVIDIA, analyzing a TensorFlow deep learning framework, and optimizing and deploying acceleration aiming at the NVIDIA equipment;
②, configuring the PB model, copying the PB model, the cable map file and the model configuration file to a data directory of TensorRT;
③, adding support to the newly added PB model, modifying the model and the model configuration file path thereof into the newly added PB model and the model configuration file path thereof under the utilis directory of TensorRT, and realizing the support to the newly added PB model;
opening a camera _ tf _ trt. py file in TensorRT, modifying the default model name into a newly added PB model name, and modifying the able map file path into a newly added able map file path;
⑤ configuring a serial port to realize recognition result output, modifying the result output part of the visualization. py file under the utils directory of TensorRT, increasing serial port data output, and realizing the serial port output of the recognition result;
step two, after the training is finished, garbage classification is started:
① initializing the garbage classification system;
②, garbage is thrown in, a garbage tray runs behind the conveyor belt and starts the conveyor belt, a user throws the garbage into the garbage throwing port, the garbage falls onto the conveyor belt, runs on the conveyor belt, falls into the garbage tray and is detected by a first photoelectric switch sensor, and at the moment, the STM32 controller controls the conveyor belt to stop;
③ garbage image collection, wherein the light sensor detects the illumination intensity and sends the illumination intensity information to the STM32 controller, the STM32 controller controls the on-off of the light supplement lamp and adjusts the illumination intensity according to the illumination intensity information to provide proper illumination intensity for the camera to collect garbage images;
④, recognizing garbage, sending the acquired garbage image to an NVIDIA machine learning platform by the camera for recognition, and sending the recognition result to an STM32 controller by the NVIDIA machine learning platform;
⑤, sorting and putting garbage, enabling an STM32 controller to sort the garbage according to the recognition result and control a sliding rail to move a garbage tray to the position above an intelligent garbage can with corresponding classification, meanwhile, enabling an STM32 controller to detect the capacity condition of the intelligent garbage can through a second photoelectric switch sensor, stopping dumping and manually cleaning if the intelligent garbage can is full, and enabling the garbage tray to dump the garbage into the corresponding intelligent garbage can if the intelligent garbage can is not full.
Compared with the prior art, the invention has the beneficial effects that:
(1) the system applies the AI technology to garbage recognition, applies the deep learning visual classification technology to garbage sorting, collects garbage images through the camera, adopts a TensorFlow deep learning framework, and greatly improves the accuracy of garbage recognition through the migration training of a MobileNet SSD model. The STM32 singlechip is handled the identification result, and the control conveyer belt realizes rubbish separation, and the control slide rail has realized the transport and the input of rubbish with the rubbish accuracy in dropping into corresponding intelligent garbage bin, need not artifical the participation, has improved work efficiency and rate of accuracy, has greatly reduced the human cost.
(2) The intelligent garbage can is provided with a second photoelectric switch sensor for detecting the capacity of the garbage can, and is provided with a garbage solid-liquid separation device and a garbage leachate recovery device, so that the environmental pollution is effectively reduced.
(3) Based on the TensorFlow deep learning framework, the garbage classification recognition method realizes learning in work, the recognized garbage types are gradually increased, and the accuracy is gradually improved.
(4) The system adopts the MoblieNetSSD model which is easier to operate on the embedded equipment and has higher recognition speed, the calculation amount is smaller, the performance is higher, and the garbage can be recognized quickly and accurately.
(5) The NVIDIA machine learning platform adopts open source codes, and later-stage upgrading and secondary development of users are facilitated. Training picture sets and picture training amount can be automatically increased, and the garbage recognition type and the garbage recognition amount can be conveniently increased in the later period of a user.
Drawings
Fig. 1 is a schematic system structure according to an embodiment of the present invention.
Fig. 2 is a structural diagram of an intelligent trash can according to an embodiment of the invention.
Fig. 3 is a recognition rate graph of a model trained using an image dataset according to embodiment 1 of the present invention.
Fig. 4 is a graph of the total loss rate of a model trained using an image dataset according to embodiment 1 of the present invention.
Fig. 5 is a graph showing the recognition effect of the image data set on the model after 11026 times of training in embodiment 1 of the present invention.
In the figure: 1. a garbage throwing port; 2. a conveyor belt; 3. a first photoelectric switch sensor; 4. a trash tray; 5. a slide rail; 6. a camera; 7. a light supplement lamp; 8. a second photoelectric switch sensor; 9. an intelligent trash can; 10. NVIDIA machine learning platform; 11. an STM32 controller; 12. a trash receiving area; 13. a trash recognition area; 14. a waste dumping zone; 15. a light sensor; 16. a support; 91. a trash can body; 92. a solid-liquid separation device for garbage; 93. garbage leachate recovery device.
Detailed Description
The present invention will be further described with reference to the following examples and accompanying drawings. The specific examples are only intended to illustrate the invention in further detail and do not limit the scope of protection of the claims of the present application.
The invention provides a visual garbage classification system (for short, see fig. 1-2) based on TensorFlow, which is characterized by comprising a garbage throwing port 1, a conveyor belt 2, a first photoelectric switch sensor 3, a garbage tray 4, a slide rail 5, a camera 6, a light supplementing lamp 7, a second photoelectric switch sensor 8, an intelligent garbage can 9, an NVIDIA machine learning platform 10, an ST M32 controller 11 and a light sensor 15;
the slide rail 5 is arranged on the ground; the garbage tray 4 is arranged on the slide rail 5, and the garbage tray 4 moves along the slide rail 5 through the slide rail 5; one end of the conveyor belt 2 is connected with the garbage throwing-in port 1, the garbage throwing-in port 1 is used for users to throw in garbage onto the conveyor belt 2, the other end of the conveyor belt is connected with the garbage tray 4, and the conveyor belt 2 is used for realizing garbage separation; the first photoelectric switch sensor 3 is arranged at the edge of one end, connected with the garbage tray 4, of the conveyor belt 2 and is used for detecting whether garbage falls to the garbage tray 4 from the conveyor belt 2 or not, and the first photoelectric switch sensor 3 is triggered when the garbage falls; the camera 6 is placed in front of the intelligent garbage can 9 through a support 16, and the camera 6 is positioned right above the garbage tray 4 which runs to the position and is used for collecting garbage images; the light supplement lamp 7 is arranged on the bracket 16 and is positioned above the camera 6 to provide sufficient illumination for the camera 6 to collect garbage images; the light sensor 15 is arranged on the support 16 and located in the illumination range of the light supplement lamp 7, the light sensor 15 detects illumination intensity and sends illumination intensity information to the STM32 controller 11, the STM32 controller 11 controls the on-off of the light supplement lamp 7 and adjusts illumination intensity according to the illumination intensity information, appropriate illumination intensity is provided for the camera 6 to collect garbage images, and the system is guaranteed to operate in all weather; a second photoelectric switch sensor 8 is mounted at the position of the opening of each intelligent garbage can 9, and the second photoelectric switch sensor 8 is used for detecting the capacity condition of the intelligent garbage can 9; the NVIDIA machine learning platform 10 is used for image recognition; the STM32 controller 11 is used for processing the image recognition result of the NVIDIA machine learning platform 10 and controlling the movement of the conveyor belt 2 and the slide rail 5;
the conveyor belt 2, the first photoelectric switch sensor 3, the slide rail 5, the light supplementing lamp 7, the second photoelectric switch sensor 8 and the light sensor 15 are all connected with the STM32 controller 11 through leads; the STM32 controller 11 is connected with the NVIDIA machine learning platform 10 through a serial port line; the camera 6 is connected with the NVIDIA machine learning platform 10 through a USB cable.
The four intelligent garbage cans 9 are arranged in the sequence of 'recoverable garbage', 'harmful garbage', 'kitchen garbage' and 'other garbage', and the four types of garbage can accurately classify the four types of garbage according with the national standard. The intelligent garbage can 9 comprises a garbage can body 91, a garbage solid-liquid separation device 92 and a garbage leachate recovery device 93; the garbage leachate recycling device 93 adopts a leachate guide pipe, is arranged at the bottom of the side wall of the garbage can body 91 and is used for collecting garbage leachate so as to avoid environmental pollution; the garbage solid-liquid separation device 92 adopts a seepage filter screen, is placed at the bottom of the garbage can body 91 and is positioned above the garbage seepage recovery device 93, and is used for separating garbage from garbage seepage;
preferably, the three areas in which the waste tray 4 operates are a waste receiving area 12, a waste identification area 13 and a waste dumping area 14; the garbage recognition area 13 is positioned between the garbage receiving area 12 and the garbage dumping area 14; the garbage receiving area 12 is arranged at the tail end of the conveyor belt 2, the camera 6 is placed in the garbage identification area 13, and the intelligent garbage can 9 is placed in the garbage dumping area 14.
The NVIDIA machine learning platform 10 needs to support TensorRT, an NVIDIA Jetson TX2 embedded machine learning platform can be adopted, wherein a neural network reasoning acceleration framework TensorRT of NVIDIA is installed to analyze a TensorFlow deep learning framework and load a MobileNet SSD model for garbage recognition;
the model of the STM32 controller 11 may employ STM32F103ZET 6;
the invention also provides a visual garbage classification method (short method) based on TensorFlow, which is characterized by comprising the following steps:
step one, training the NVIDIA machine learning platform 10;
(1) making an image recognition training set:
① collecting training pictures, collecting pictures of various garbage objects shot under different backgrounds, different lights, different angles, different garbage quantities and the like, acquired by tools such as a web crawler and the like and screened from MS COCO data set to obtain the training pictures for improving the accuracy;
preferably, the specific method for screening the pictures of various junk objects from the MS COCO dataset is as follows: screening out a corresponding object classification list from the MS COCO data set, and extracting training pictures from train and val folders of the MS COCO data set according to the classification list;
②, screening training pictures, manually screening the training pictures, and deleting partial unqualified pictures to obtain a training picture set, wherein the pictures in the training picture set must contain clear images of objects, have typical characteristics of the objects and diversify object backgrounds;
③, labeling the training pictures, adopting LabLeImg software to label the pictures in the training picture set, namely identifying the garbage in the pictures, and outputting an XML labeling file in a Pascal VOC format by the software;
④, generating a data set in a TFRecord format, generating an XML annotation file into a unified TFRecord format file by using a program create _ past _ tf _ record.py of an object detection library (Objectdetection API) of TensorFlow to obtain the data set in the TFRecord format, and finishing the production of an image recognition training set;
(2) training is started, and the training process is as follows:
① downloading MobileNet SSD pre-training model, downloading the pre-training model SSD _ MobileNet to object _ detection folder of object detection library of TensorFlow;
②, modifying the object type file, adding or deleting the object type contained in the corresponding label map file (mapping file of object label number and object name) in the object _ detection/data folder according to the actual object type of the garbage;
③ modifies object _ detection/samples/configurations model configuration file ssd _ mobilene _ v1_ coco. configuration, namely modifying object type number num _ classes according to actual training garbage type, specifying file path input _ path of training and verifying TFRecord format data set (namely training and verifying TFRecord file path), specifying LABLE map file storage path;
④ adjusting the batch size of the pre-training model to be trained according to the configuration of the computer hardware CPU and the memory, starting training, training the pre-training model to be a trained model;
⑤ converting the trained model into a PB model (PB, protocol buffer), outputting a program export _ reference _ graph through a model file of a target detection library of Tensflow, and converting the trained model into a PB model (garbage recognition model) capable of running independently;
⑥ testing the training result, testing the trained PB model through object _ detection _ configuration.py of a target detection library of TensorFlow, and increasing the number of the labeled pictures of each object (generally, each object is not less than 1000) and adjusting the parameter value in the model configuration file ssd _ mobilene _ v1_ coco.config to improve the recognition rate of the PB model to the pictures to a preset value;
(3) the specific method/steps of the operating environment configuration of the NVIDIA machine learning platform 10 are:
①, installing a neural network reasoning acceleration framework TensorRT of NVIDIA, analyzing deep learning frameworks such as TensorFlow, Caffe, PyTorch or MXnet, and optimizing and deploying acceleration aiming at the NVIDIA equipment;
② configuring PB model, copying PB model, able map file and model configuration file ssd _ mobilenet _ v1_ coco.config to data directory of TensorRT;
③, adding support to a newly added PB model, and modifying the model and the model configuration file path thereof into the newly added PB model and the model configuration file path thereof in the egowings _ models.
Opening a camera _ tf _ trt. py file in TensorRT, modifying the default model name into a newly added PB model name, and modifying the able map file path into a newly added able map file path;
⑤ configuring a serial port to realize recognition result output, modifying the result output part of the visualization. py file under the utils directory of TensorRT, increasing serial port data output, and realizing the serial port output of the recognition result;
step two, after the training is finished, starting garbage classification based on vision:
① initializing the garbage classification system, initializing the NVIDIA machine learning platform 10;
②, garbage is thrown in, the garbage tray 4 runs behind the conveyor belt 2 of the garbage receiving area 12, the conveyor belt 2 is started, a user opens a bag to throw the garbage into the garbage throwing opening 1, the garbage falls onto the conveyor belt 2 and runs all the time through the conveyor belt 2 to realize garbage separation, the garbage runs on the conveyor belt 2, falls into the garbage tray 4 and is detected by the first photoelectric switch sensor 3, and the STM32 controller 11 controls the conveyor belt 2 to stop;
③ garbage image collection, wherein the light sensor 15 detects illumination intensity and sends illumination intensity information to the STM32 controller 11, the STM32 controller 11 controls the on-off of the light supplement lamp 7 and adjusts illumination intensity according to the illumination intensity information to provide proper illumination intensity for the camera 6 to collect garbage images, and then the garbage tray 4 moves to the garbage recognition area 13 below the camera 6 to collect garbage images;
④, recognizing garbage, sending the acquired garbage image to the NVIDIA machine learning platform 10 by the camera 6 for recognition, and sending the recognition result to the STM32 controller 11 by the NVIDIA machine learning platform 10;
⑤, sorting and putting garbage, enabling an STM32 controller 11 to sort the garbage according to recognition results and control a slide rail 5 to move a garbage tray 4 to the position above an intelligent garbage can 9 of the corresponding sort, meanwhile, enabling an STM32 controller 11 to detect the capacity condition of the intelligent garbage can 9 through a second photoelectric switch sensor 8, enabling the garbage tray 4 to stop dumping if the intelligent garbage can 9 is full, enabling a loudspeaker in the STM32 controller 11 to emit a prompt sound of 'garbage can full and timely cleaning' to perform manual cleaning, enabling the loudspeaker to emit a corresponding garbage dumping prompt sound if the intelligent garbage can 9 is not full, enabling the garbage tray 4 to dump the garbage into the corresponding intelligent garbage can 9, and enabling the garbage tray 4 to return to a garbage receiving area 12 if operation is overtime.
Example 1
The system identifies and classifies harmful garbage-tablet boards, and 319 pictures are contained in an image data set of the tablet boards; as can be seen from fig. 3, after the model is trained 9817 times through the image dataset, the recognition rate (the mAP value) of the model reaches 80%. As can be seen from fig. 4, the total loss rate decreased to 10% after 11026 times of model training through the image dataset. As can be seen from fig. 5, after 11026 times of model training through the image dataset, all tablet plates can be identified.
Nothing in this specification is said to apply to the prior art.
Claims (9)
1. A visual garbage classification system based on TensorFlow is characterized by comprising a conveyor belt, a first photoelectric switch sensor, a garbage tray, a slide rail, a camera, a second photoelectric switch sensor, an intelligent garbage can, an NVIDIA machine learning platform and an STM32 controller;
the garbage tray is arranged on the sliding rail, so that the garbage tray can move along the sliding rail; one end of the conveyor belt is connected with the garbage throwing port, and the other end of the conveyor belt is connected with the garbage tray; the first photoelectric switch sensor is arranged at the edge of one end, connected with the garbage tray, of the conveyor belt and is used for detecting whether garbage falls to the garbage tray from the conveyor belt or not; the camera is placed in front of the intelligent garbage can through the bracket and used for collecting garbage images; a second photoelectric switch sensor is mounted at the position of the opening of each intelligent garbage can and used for detecting the capacity condition of the intelligent garbage can; the conveyor belt, the first photoelectric switch sensor, the slide rail and the second photoelectric switch sensor are all connected with an STM32 controller; the STM32 controller is connected with the NVIDIA machine learning platform; the camera is connected with the NVIDIA machine learning platform.
2. The TensorFlow-based visual trash classification system of claim 1, further comprising a fill light and a light sensor; the light supplement lamp is arranged on the bracket and provides sufficient illumination for the camera to collect garbage images; the light sensor is located in the illumination range of the light supplement lamp, detects illumination intensity and sends the illumination intensity information to the STM32 controller, and the STM32 controller controls the light supplement lamp to be turned on and off and adjusts illumination intensity according to the illumination intensity information to provide proper illumination intensity for the camera to collect garbage images; light filling lamp and light sensor all are connected with STM32 controller.
3. The visual garbage classification system based on TensorFlow according to claim 1, characterized in that the intelligent garbage can comprises a garbage can body, a garbage solid-liquid separation device and a garbage leachate recovery device; the garbage leachate recovery device is arranged at the bottom of the side wall of the garbage can body and used for collecting garbage leachate; the garbage solid-liquid separation device is placed at the bottom of the garbage can body and is positioned above the garbage leachate recovery device and used for separating garbage from garbage leachate.
4. The TensorFlow-based visual trash classification system of claim 3, wherein the trash leachate recovery device employs a leachate conduit; the solid-liquid separation device of the garbage adopts a seepage filter screen.
5. The TensorFlow based visual trash classification system of claim 1, wherein the three areas where the trash tray runs are a trash receiving area, a trash recognition area, and a trash dumping area, respectively; the garbage identification area is positioned between the garbage receiving area and the garbage dumping area; the rubbish receiving area sets up at the conveyer belt end, and the camera is placed in rubbish discernment district, and intelligent garbage bin is placed in rubbish district of dumping.
6. A garbage classification method of the TensorFlow based visual garbage classification system according to any one of claims 1 to 5, characterized in that the method comprises the following steps:
step one, training an NVIDIA machine learning platform;
(1) making an image recognition training set:
① collecting training pictures, collecting pictures of various garbage objects to obtain training pictures;
② screening training pictures, manually screening the training pictures to obtain a training picture set, wherein the pictures in the training picture set contain clear images of objects, have typical characteristics of the objects and diversify object backgrounds;
③ marking training pictures, marking pictures in the training picture set, identifying garbage in the pictures, and outputting the pictures as XML marking files;
④, generating a data set in a TFrecord format, and generating an XML label file into a uniform TFrecord format file by using a program of a target detection library of TensorFlow to obtain the data set in the TFrecord format;
(2) training is started, and the training process is as follows:
① downloading the pre-training model to the object _ detection folder of the object detection library of Tensflow;
②, modifying the object type file, adding or deleting the object type contained in the corresponding layout map file in the object _ detection/data folder according to the actual object type of the garbage;
③ modifying the model configuration file in object _ detection/samples/configurations, modifying the object type quantity according to the actual training garbage type, specifying the file path of the data set in TFRecord format for training and verification, specifying the storage path of the table map file;
④ adjusting the batch size of the pre-training model according to the configuration of the computer hardware CPU and the memory, starting training, training the pre-training model into a trained model;
⑤ converting the trained model into PB model which can run independently through the model file output program of the object detection library of TensorFlow;
⑥ testing the training result, testing the PB model by a testing program of a target detection library of TensorFlow, and increasing the number of the labeled pictures of each object and adjusting the parameter values in the model configuration file to ensure that the recognition rate of the PB model to the pictures is increased to a preset value;
(3) the specific steps of the running environment configuration of the NVIDIA machine learning platform are as follows:
①, installing a neural network reasoning acceleration framework TensorRT of NVIDIA, analyzing a TensorFlow deep learning framework, and optimizing and deploying acceleration aiming at the NVIDIA equipment;
②, configuring the PB model, copying the PB model, the able map file and the model configuration file to a dat a directory of TensorRT;
③, adding support to the newly added PB model, modifying the model and the model configuration file path thereof into the newly added PB model and the model configuration file path thereof under the utilis directory of TensorRT, and realizing the support to the newly added PB model;
opening a camera _ tf _ trt. py file in TensorRT, modifying the default model name into a newly added PB model name, and modifying the able map file path into a newly added able map file path;
⑤ configuring a serial port to realize recognition result output, modifying the result output part of the visualization. py file under the utils directory of TensorRT, increasing serial port data output, and realizing the serial port output of the recognition result;
step two, after the training is finished, garbage classification is started:
① initializing the garbage classification system;
②, garbage is thrown in, a garbage tray runs behind the conveyor belt and starts the conveyor belt, a user throws the garbage into the garbage throwing port, the garbage falls onto the conveyor belt, runs on the conveyor belt, falls into the garbage tray and is detected by a first photoelectric switch sensor, and at the moment, the STM32 controller controls the conveyor belt to stop;
③ garbage image collection, wherein the light sensor detects the illumination intensity and sends the illumination intensity information to the STM32 controller, the STM32 controller controls the on-off of the light supplement lamp and adjusts the illumination intensity according to the illumination intensity information to provide proper illumination intensity for the camera to collect garbage images;
④, recognizing garbage, sending the acquired garbage image to an NVIDIA machine learning platform by the camera for recognition, and sending the recognition result to an STM32 controller by the NVIDIA machine learning platform;
⑤, sorting and putting garbage, enabling an STM32 controller to sort the garbage according to the recognition result and control a sliding rail to move a garbage tray to the position above an intelligent garbage can with corresponding classification, meanwhile, enabling an STM32 controller to detect the capacity condition of the intelligent garbage can through a second photoelectric switch sensor, stopping dumping and manually cleaning if the intelligent garbage can is full, and enabling the garbage tray to dump the garbage into the corresponding intelligent garbage can if the intelligent garbage can is not full.
7. The method of claim 6, wherein in step one, pictures of various spam objects are taken at different times, obtained through a network, and filtered from the MS COCO data set.
8. The method for garbage classification according to claim 7, wherein in the first step, the specific method for screening the images of various garbage objects from the MS COCO data set is as follows: and screening out a corresponding object classification list from the MS COCO data set, and extracting training pictures from the train and val folders of the MS COCO data set according to the classification list.
9. The garbage classification method according to the claim 6, wherein in the second step, the STM32 controller detects the capacity condition of the intelligent garbage can through the second photoelectric switch sensor; if the intelligent garbage can is full, the garbage tray stops dumping, and a loudspeaker in the STM32 controller emits corresponding prompt tones to perform manual cleaning; if the intelligent garbage can is not full, the loudspeaker emits corresponding garbage dumping prompt sound, and the garbage tray dumps the garbage into the corresponding intelligent garbage can; if the operation is overtime, the garbage tray returns to the garbage receiving area.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911189575.1A CN110884791A (en) | 2019-11-28 | 2019-11-28 | Vision garbage classification system and classification method based on TensorFlow |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911189575.1A CN110884791A (en) | 2019-11-28 | 2019-11-28 | Vision garbage classification system and classification method based on TensorFlow |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110884791A true CN110884791A (en) | 2020-03-17 |
Family
ID=69749205
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911189575.1A Pending CN110884791A (en) | 2019-11-28 | 2019-11-28 | Vision garbage classification system and classification method based on TensorFlow |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110884791A (en) |
Cited By (23)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111498326A (en) * | 2020-04-03 | 2020-08-07 | 成都禧来科技有限公司 | Automatic garbage classifier based on two-class recognition model |
| CN111517018A (en) * | 2020-04-26 | 2020-08-11 | 佛山科学技术学院 | Intelligent garbage classification device and classification method |
| CN111619992A (en) * | 2020-06-11 | 2020-09-04 | 中南林业科技大学 | Intelligent garbage classification system and method based on machine vision |
| CN111767804A (en) * | 2020-06-08 | 2020-10-13 | 上海交通大学 | Recyclable garbage image classification method and system based on artificial intelligence |
| CN111914773A (en) * | 2020-08-07 | 2020-11-10 | 杭州微胜智能科技有限公司 | Equipment and method for capturing illegal boarding and alighting of passengers |
| CN111974704A (en) * | 2020-08-14 | 2020-11-24 | 东北大学秦皇岛分校 | Garbage classification detection system and method based on computer vision |
| CN112007933A (en) * | 2020-08-15 | 2020-12-01 | 北京农学院 | Portable quick recovery processing system based on gardens discarded object |
| CN112027424A (en) * | 2020-09-23 | 2020-12-04 | 深圳市康灵盾智能厨具设备有限公司 | Garbage classification and recovery system and garbage classification and recovery method |
| CN112407680A (en) * | 2020-11-06 | 2021-02-26 | 钱继红 | Intensive intelligent garbage collection equipment and working method thereof |
| CN112429425A (en) * | 2020-11-30 | 2021-03-02 | 湖北文理学院 | Garbage classification device |
| CN112455614A (en) * | 2020-11-24 | 2021-03-09 | 武汉理工大学 | Water surface garbage collection and classification device, control method and garbage cleaning ship |
| CN112488167A (en) * | 2020-11-19 | 2021-03-12 | 安徽理工大学 | Rubbish identification and classification system based on improved EfficientNet network |
| CN112597919A (en) * | 2020-12-28 | 2021-04-02 | 浙江工业大学 | Real-time medicine box detection method based on YOLOv3 pruning network and embedded development board |
| CN112693767A (en) * | 2020-12-14 | 2021-04-23 | 秦郡酉 | Garbage classification robot |
| CN112733676A (en) * | 2020-12-31 | 2021-04-30 | 青岛海纳云科技控股有限公司 | Method for detecting and identifying garbage in elevator based on deep learning |
| CN113000547A (en) * | 2021-01-25 | 2021-06-22 | 福建名盛美洁环境工程有限公司 | River course rubbish cleans intelligent classification recovery unit |
| CN113627830A (en) * | 2021-09-08 | 2021-11-09 | 西安智者云集云计算有限公司 | Scrap steel grading system and method |
| CN114013856A (en) * | 2021-11-05 | 2022-02-08 | 东北林业大学 | Garbage intelligent recognition automatic classification recovery device and real-time metering monitoring system thereof |
| CN114261658A (en) * | 2022-01-06 | 2022-04-01 | 浙江金实乐环境工程有限公司 | Guide garbage truck with automatic garbage throwing and classifying functions |
| CN114355907A (en) * | 2021-12-22 | 2022-04-15 | 东风汽车集团股份有限公司 | Cloud-based intelligent garbage recognition and cleaning method and system |
| CN116197210A (en) * | 2022-09-06 | 2023-06-02 | 中科澳森机器人科技(南京)有限公司 | Automatic household garbage classification integrated equipment based on artificial intelligence |
| CN116894987A (en) * | 2023-09-11 | 2023-10-17 | 南京信息工程大学 | Recycle degree assessment and classification method and device based on multi-sensing characteristics |
| CN117152521A (en) * | 2023-09-04 | 2023-12-01 | 长江大学 | Garbage can device based on convolutional neural network and embedded technology and use method |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2012104441A1 (en) * | 2010-12-09 | 2012-08-09 | Easyrecycling S L | Waste-sorting street bin for recycling |
| CN108197633A (en) * | 2017-11-24 | 2018-06-22 | 百年金海科技有限公司 | Deep learning image classification based on TensorFlow is with applying dispositions method |
| CN110119662A (en) * | 2018-03-29 | 2019-08-13 | 王胜春 | A kind of rubbish category identification system based on deep learning |
| CN110222680A (en) * | 2019-05-19 | 2019-09-10 | 天津大学 | A kind of domestic waste article outer packing Method for text detection |
| CN110254997A (en) * | 2019-06-10 | 2019-09-20 | 徐娇娇 | Automatic classification dustbin |
-
2019
- 2019-11-28 CN CN201911189575.1A patent/CN110884791A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2012104441A1 (en) * | 2010-12-09 | 2012-08-09 | Easyrecycling S L | Waste-sorting street bin for recycling |
| CN108197633A (en) * | 2017-11-24 | 2018-06-22 | 百年金海科技有限公司 | Deep learning image classification based on TensorFlow is with applying dispositions method |
| CN110119662A (en) * | 2018-03-29 | 2019-08-13 | 王胜春 | A kind of rubbish category identification system based on deep learning |
| CN110222680A (en) * | 2019-05-19 | 2019-09-10 | 天津大学 | A kind of domestic waste article outer packing Method for text detection |
| CN110254997A (en) * | 2019-06-10 | 2019-09-20 | 徐娇娇 | Automatic classification dustbin |
Non-Patent Citations (1)
| Title |
|---|
| 廖义奎: "《物联网移动软件开发》", 30 September 2019, 北京航空航天大学出版社 * |
Cited By (27)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111498326A (en) * | 2020-04-03 | 2020-08-07 | 成都禧来科技有限公司 | Automatic garbage classifier based on two-class recognition model |
| CN111517018A (en) * | 2020-04-26 | 2020-08-11 | 佛山科学技术学院 | Intelligent garbage classification device and classification method |
| CN111767804A (en) * | 2020-06-08 | 2020-10-13 | 上海交通大学 | Recyclable garbage image classification method and system based on artificial intelligence |
| CN111619992A (en) * | 2020-06-11 | 2020-09-04 | 中南林业科技大学 | Intelligent garbage classification system and method based on machine vision |
| CN111914773A (en) * | 2020-08-07 | 2020-11-10 | 杭州微胜智能科技有限公司 | Equipment and method for capturing illegal boarding and alighting of passengers |
| CN111974704A (en) * | 2020-08-14 | 2020-11-24 | 东北大学秦皇岛分校 | Garbage classification detection system and method based on computer vision |
| CN112007933A (en) * | 2020-08-15 | 2020-12-01 | 北京农学院 | Portable quick recovery processing system based on gardens discarded object |
| CN112027424A (en) * | 2020-09-23 | 2020-12-04 | 深圳市康灵盾智能厨具设备有限公司 | Garbage classification and recovery system and garbage classification and recovery method |
| CN112407680A (en) * | 2020-11-06 | 2021-02-26 | 钱继红 | Intensive intelligent garbage collection equipment and working method thereof |
| CN112488167A (en) * | 2020-11-19 | 2021-03-12 | 安徽理工大学 | Rubbish identification and classification system based on improved EfficientNet network |
| CN112455614A (en) * | 2020-11-24 | 2021-03-09 | 武汉理工大学 | Water surface garbage collection and classification device, control method and garbage cleaning ship |
| CN112429425A (en) * | 2020-11-30 | 2021-03-02 | 湖北文理学院 | Garbage classification device |
| CN112693767A (en) * | 2020-12-14 | 2021-04-23 | 秦郡酉 | Garbage classification robot |
| CN112597919A (en) * | 2020-12-28 | 2021-04-02 | 浙江工业大学 | Real-time medicine box detection method based on YOLOv3 pruning network and embedded development board |
| CN112733676A (en) * | 2020-12-31 | 2021-04-30 | 青岛海纳云科技控股有限公司 | Method for detecting and identifying garbage in elevator based on deep learning |
| CN112733676B (en) * | 2020-12-31 | 2024-09-24 | 海纳云物联科技有限公司 | Deep learning-based method for detecting and identifying garbage in elevator |
| CN113000547B (en) * | 2021-01-25 | 2022-06-07 | 福建名盛美洁环境工程有限公司 | River course rubbish cleans intelligent classification recovery unit |
| CN113000547A (en) * | 2021-01-25 | 2021-06-22 | 福建名盛美洁环境工程有限公司 | River course rubbish cleans intelligent classification recovery unit |
| CN113627830A (en) * | 2021-09-08 | 2021-11-09 | 西安智者云集云计算有限公司 | Scrap steel grading system and method |
| CN114013856A (en) * | 2021-11-05 | 2022-02-08 | 东北林业大学 | Garbage intelligent recognition automatic classification recovery device and real-time metering monitoring system thereof |
| CN114355907A (en) * | 2021-12-22 | 2022-04-15 | 东风汽车集团股份有限公司 | Cloud-based intelligent garbage recognition and cleaning method and system |
| CN114355907B (en) * | 2021-12-22 | 2024-01-19 | 东风汽车集团股份有限公司 | Cloud-based intelligent garbage identification and cleaning method and system |
| CN114261658A (en) * | 2022-01-06 | 2022-04-01 | 浙江金实乐环境工程有限公司 | Guide garbage truck with automatic garbage throwing and classifying functions |
| CN116197210A (en) * | 2022-09-06 | 2023-06-02 | 中科澳森机器人科技(南京)有限公司 | Automatic household garbage classification integrated equipment based on artificial intelligence |
| CN117152521A (en) * | 2023-09-04 | 2023-12-01 | 长江大学 | Garbage can device based on convolutional neural network and embedded technology and use method |
| CN116894987A (en) * | 2023-09-11 | 2023-10-17 | 南京信息工程大学 | Recycle degree assessment and classification method and device based on multi-sensing characteristics |
| CN116894987B (en) * | 2023-09-11 | 2023-11-21 | 南京信息工程大学 | Recycle degree assessment and classification method and device based on multi-sensing characteristics |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110884791A (en) | Vision garbage classification system and classification method based on TensorFlow | |
| CN110861852A (en) | Automatic classification garbage can based on visual identification and classification method | |
| CN211033861U (en) | Automatic garbage classification device | |
| CN110861854B (en) | Intelligent garbage classification device and intelligent garbage classification method | |
| CN205708275U (en) | A kind of automatic sorting refuse bin | |
| CN203782622U (en) | Intelligent garbage-collecting-sorting educating robot | |
| CN110355188B (en) | Smart garbage classification in-situ reduction recycling system based on multiple land utilization | |
| CN107600791A (en) | Interactive automatic garbage classification collection box | |
| CN110076782A (en) | A kind of intelligence control garbage classification machine people of computer vision target identification | |
| CN204751110U (en) | Hierarchical automatic classification dustbin | |
| CN201997485U (en) | Garbage auto-separation device | |
| CN206798308U (en) | A kind of automatic recovery rewarding device of refuse classification detection | |
| CN113213005A (en) | Automatic change refuse classification processing apparatus | |
| CN110902216A (en) | Multiple categorised rubbish transfer device | |
| CN111137602B (en) | Garbage classification method and garbage classification system based on artificial intelligence | |
| CN109878946A (en) | A kind of automatic recovery and processing system of city distribution rubbish | |
| CN109625681A (en) | A kind of dustbin of automatic identification rubbish classification | |
| CN112090925A (en) | Garbage treatment method and device with broken bags at rear ends | |
| CN116620744A (en) | Intelligent classification system based on deep learning and implementation method thereof | |
| CN206701918U (en) | A kind of garbage sorting device of Multi-sensor Fusion | |
| CN210417814U (en) | Automatic garbage sorting device | |
| CN201520512U (en) | Automatic classification garbage deployment device | |
| CN111361889A (en) | Intelligent garbage can and control method thereof | |
| CN205868755U (en) | But automatic classification's waste classification is quick -witted | |
| CN210853828U (en) | Wet garbage collection equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200317 |
|
| RJ01 | Rejection of invention patent application after publication |