Multi-task garbage intelligent sorting device based on deep learning and movable mechanical claws
Technical Field
The invention relates to the technical field of garbage classification, in particular to a multi-task intelligent garbage sorting device based on deep learning and mobile mechanical claws.
Background
Today, environmental problems develop gradually, environmental pollution is aggravated gradually, and global attention is attracted. Improper disposal of waste is one of the main factors causing environmental pollution, and ordered waste classification becomes the primary task to solve such pollution. In recent years, china is gradually pushing garbage classification all over the country, good effects are shown, 46 main cities are tested in the early stage, and positive development of garbage classification is promoted.
The garbage classification can help us maximize the utilization of garbage resources, reduce the total amount of garbage, improve our living environment, reduce the pollution of garbage to underground water, and reduce the occupied area of garbage. Many waste contains substances that are difficult to decompose naturally, which are very harmful to the land, while waste classification enables the identification and individual disposal of these recoverable or difficult to degrade substances, thus reducing the amount of waste by at least 60% or more.
Therefore, garbage classification is particularly important as a primary link in garbage disposal flow, and intelligent garbage classification becomes a research hotspot. Traditional intelligent garbage classification device mostly can satisfy rubbish and drops into one by one, then through the technique of computer vision detection, accomplishes the autonomic judgement and the classified storage to various rubbish, to the condition that a plurality of different kinds of rubbish were mixed together drops into the garbage bin, traditional intelligent garbage bin's design can not carry out intelligent categorised delivery and storage, has influenced holistic letter sorting efficiency.
Disclosure of Invention
Aiming at the defects of the prior art, the invention provides the intelligent multi-task garbage sorting device based on deep learning and mobile mechanical claws, which realizes sorting delivery and storage of a plurality of different types of garbage.
The intelligent multi-task garbage sorting device comprises a sorting box body, garbage sorting equipment, a camera, a development board and a controller, wherein the top end of the sorting box body is provided with a garbage throwing port, the garbage sorting equipment, the camera, the development board and the controller are arranged in the box body, the camera detects garbage images thrown in through the garbage throwing port in real time and transmits the garbage images to the development board, the development board recognizes and gives corresponding garbage types and coordinate positions through a visual algorithm and transmits the garbage types and coordinate positions to the controller, and the controller controls the garbage sorting equipment to throw in garbage cans corresponding to the lower portion of the box body according to received garbage coordinate values.
Preferably, the device further comprises a detection platform and a tray arranged in the sorting box body; the detection platform is positioned at the lower part of the garbage throwing opening and consists of an acrylic plate with an opening and a closing middle, two ends of the acrylic plate are respectively fixed on an optical axis, the optical axis is controlled to rotate through a first steering engine to realize the opening and the closing of the detection platform, the tray is arranged below the detection platform and controls the rotating angle through a second steering engine, and the first steering engine and the second steering engine control the rotating angle through a controller.
The STM32 controller drives the stepping motor to control the two-dimensional sliding table and the mechanical claw to move to the coordinate position of the garbage when receiving the coordinate value of the current garbage, starts the mechanical claw to grab the garbage and returns to the central position of the detection platform, at the moment, the first steering engine for controlling the opening and closing of the detection platform rotates for 90 degrees, the detection platform is opened, the garbage falls into a tray controlled by a second steering engine below, and the second steering engine controls the tray to rotate to the position corresponding to the garbage can, so that the garbage can be accurately classified.
Preferably, the device further comprises a display screen arranged at the top of the sorting box body and used for displaying sorting and delivering results of various garbage in real time.
Preferably, the development board deploys a visual algorithm to calculate the object type and specific coordinate value of the identified garbage, and feeds back the coordinate value to the controller, and the controller controls the stepping motor to realize the movement of the two-dimensional sliding table, so that the mechanical gripper is positioned to the garbage with the specific coordinate value, and the garbage is grabbed.
Preferably, the specific method for calculating the specific coordinate value of the identified garbage by using the development board deployment visual algorithm comprises the following steps:
step 1, constructing a garbage classification detection model based on YOLOv and a normalized attention module;
Adopting YOLOv network model and normalized attention module as main frame of garbage classification detection model, wherein YOLOv network model is divided into 4 parts, namely main feature extraction network, feature fusion network, DBL convolution layer and loss function, wherein in YOLOv network, a normalized attention module is embedded at the end of each network block to make the network lighter and improve detection efficiency;
Step2, counting common household garbage types and constructing a training data set training garbage classification detection model;
in order to realize the accurate classification of multiple types of garbage, a multi-classification model is trained, and the accurate classification of the multiple types of garbage is realized;
step 3, identifying and classifying the garbage put on the detection platform by using a trained garbage classification detection model, calculating the coordinate position of the garbage put on the detection platform and marking the coordinate position;
and 4, returning the relative coordinates of the targets within the detection range of the camera by utilizing the target detection characteristics of the YOLO model, acquiring depth information by measuring the distance between the camera and the detection platform, mapping the coordinates of the targets acquired by the camera to the coordinates of the detection targets in the world coordinate system by vector transformation, and converting the coordinates of the detection targets in the world coordinate system into the coordinates of the targets of the two-dimensional sliding table by proportional operation, so as to realize accurate grabbing of the target garbage.
Preferably, the development board sends the identified garbage category and the coordinate position to the controller through a serial port protocol, the controller drives the stepping motor to move by using an acceleration and deceleration algorithm after receiving corresponding information and moves to the coordinate position of the corresponding garbage, the controller drives the steering engine to drive the mechanical gripper to move, the target garbage is grabbed, the target garbage returns to the middle position of the detection platform after being grabbed, the garbage is dropped and the detection platform is started, and the garbage falls into a tray controlled by the steering engine, so that the garbage is put into the specified garbage can.
Preferably, the device further comprises a diffuse reflection photoelectric switch for detecting whether the garbage can is full, the diffuse reflection photoelectric switch is controlled by the controller, the photoelectric switch is arranged at the position of each garbage can at a fixed height, when garbage in the garbage can reaches the height, the voice broadcasting module is started to carry out full-load prompt, corresponding garbage can full-load display is given on the display screen, and further full-load judgment of the garbage can is realized.
The technical proposal has the advantages that the intelligent multi-task garbage sorting device based on the deep learning and the moving mechanical claw,
(1) By using the improved YOLOv5+NAM attention module, the network is lighter, the detection efficiency and the anti-interference capability are further improved, and the detection robustness and accuracy are higher;
(2) An electric control algorithm is improved, and an acceleration and deceleration algorithm is used for driving a stepping motor, so that the garbage throwing work after clamping is more stable;
(3) The problems of classification, identification and delivery of various garbage are solved by combining software and hardware, adding target positioning and matching with a mechanical gripper.
Drawings
Fig. 1 is a schematic structural diagram of a multi-task intelligent garbage sorting device based on deep learning and moving mechanical claws, which is provided by the embodiment of the invention;
fig. 2 is a flow chart of intelligent garbage sorting by the multi-task intelligent garbage sorting device based on deep learning and moving mechanical claws, which is provided by the embodiment of the invention;
Fig. 3 is a schematic diagram of diffuse reflection photoelectric switch detection provided in an embodiment of the present invention, where (a) is a schematic diagram of diffuse reflection switch installation, and (b) is a schematic diagram of full load detection;
FIG. 4 is a diagram illustrating a YOLOV5+NAM model architecture for an improvement provided by an embodiment of the present invention;
FIG. 5 is a schematic diagram of a CBAM-channel attention sub-module provided by an embodiment of the present invention;
FIG. 6 is a schematic diagram of a SAM space attention sub-module according to an embodiment of the present invention;
FIG. 7 is a schematic diagram of a network model YOLOV of an improved portion according to an embodiment of the present invention;
FIG. 8 is a diagram of automatic labeling results of targets on an original image according to an embodiment of the present invention;
fig. 9 is a schematic diagram of a back-deduced relative position coordinate according to an embodiment of the present invention.
In the figure, 1, a camera; 2, a highlight display screen, 3, a mechanical gripper, 4, a second steering engine, 5, a first steering engine, 6, a garbage can, 7, an acrylic plate, 8, a stepping motor, 9, an aluminum profile, 10, a two-dimensional sliding table, 11 and a detection platform.
Detailed Description
The following describes in further detail the embodiments of the present invention with reference to the drawings and examples. The following examples are illustrative of the invention and are not intended to limit the scope of the invention.
In the embodiment, the intelligent multi-task garbage sorting device based on deep learning and moving mechanical claws is shown in fig. 1 and 2, and comprises a sorting box body with a garbage throwing port at the top end, a detection platform 11, a tray, a first steering engine 5, a second steering engine 4, garbage sorting equipment, cameras 1, jetson nano development boards and an STM32 controller, wherein the detection platform 11, the cameras 1 detect garbage images thrown through the garbage throwing port in real time and transmit the garbage images to the jetson nano development boards, the jetson nano development boards recognize and give out coordinate positions of corresponding garbage through visual algorithms, the recognized garbage types and the coordinate positions are sent to the STM32 controller through serial port protocols, the detection platform 11 is positioned at the lower part of the garbage throwing port and is composed of an acrylic plate 7 with an opening and closing middle, two ends of the acrylic plate 7 are respectively fixed on an optical axis, the optical axis is controlled to rotate through the first steering engine 5, the opening and closing of the detection platform 11 are realized, the tray is arranged below the detection platform 11, the rotation angle is controlled through the second steering engine 4, and the first steering engine 5 and the second steering engine 4 control the rotation angle is controlled through the controller.
The garbage sorting equipment comprises a two-dimensional sliding table 10, a flexible mechanical claw 3 and a stepping motor 8, when the STM32 controller receives the coordinate value of the current garbage, the stepping motor 8 is driven by an acceleration and deceleration algorithm to control the two-dimensional sliding table 10 and the mechanical claw 3 to move to the coordinate position of the garbage, the mechanical claw 3 is started to grab garbage and return to the central position of the detection platform 11, at the moment, the detection platform 11 is controlled to be opened and closed by a first steering engine 5 to rotate for 90 degrees, the detection platform 11 is opened (only an acrylic plate on one side is controlled to be opened), the garbage falls into a tray controlled by a second steering engine 4 below, and the second steering engine 4 controls the tray to rotate to the position corresponding to the garbage can, so that the garbage can be accurately sorted.
Meanwhile, the device also comprises a highlight display screen 2 arranged at the top of the sorting box body and diffuse reflection photoelectric switches arranged on the inner walls of the garbage cans, as shown in fig. 3, wherein the display screen 2 is used for displaying sorting and delivering results of various garbage in real time. The diffuse reflection photoelectric switch is controlled by the STM32 singlechip to detect whether the dustbin is full. In this embodiment, this photoelectric switch is installed in four garbage bins high 80%'s position, and when the height of rubbish reaches 80% in having the garbage bin, start voice broadcast module and carry out full load suggestion to give corresponding garbage bin full load display on display screen 2, and then judge whether there is the garbage bin full load.
In this embodiment, the letter sorting case box adopts 4040 aluminium alloy 9 as the frame, and the link between the aluminium alloy 9 adopts M5 bolt and ship type nut to fasten, and the device is all around and the top adopts black acrylic board to encapsulate, and a high-brightness display screen 2 is placed at the box top and is used for showing the real-time classification and the delivery result of all kinds of rubbish, designs 200 at the top of box and puts in mouthful for the delivery of multitasking rubbish. When a plurality of garbage is input into the detection platform 11 through the input port, the camera 1 at the top of the box detects the garbage, the jetson nano development board accurately identifies the garbage type through a visual algorithm and gives the coordinate position of the corresponding garbage, the STM32 controller drives the stepping motor 8 to control the two-dimensional sliding table 10 and the mechanical claw 3 to move to the coordinate position of the garbage by using an acceleration-deceleration algorithm when receiving the coordinate value of the current garbage, the mechanical claw 3 is started to grab, the garbage is grabbed and returned to the central position of the detection platform 10, the first steering engine 5 controlling the opening and closing of the detection platform 10 is rotated for 90 degrees at the moment, the detection platform 11 is opened, the garbage falls into a tray controlled by the lower steering engine, so that the garbage falls into the corresponding garbage can, and if the camera 1 detects the garbage on the platform 11, the mechanical claw 3 returns to the coordinate position of the next garbage, grabs and inputs the garbage into the corresponding garbage can until the garbage on the detection platform is sorted, and intelligent classification and delivery of the multi-task garbage are realized. In the embodiment, four sub-garbage cans 6 with the size of 200mm long by 200mm wide by 350mm high are designed at the bottom of the box body and are respectively used for storing recyclable garbage, kitchen garbage, harmful garbage and other garbage, diffuse reflection photoelectric switches are arranged at the positions of 80% of the four garbage cans, so that full load detection of the garbage cans is realized, and when the garbage cans are full load, corresponding voice prompts are given and corresponding information such as full load of the garbage cans is displayed on a screen;
In the embodiment, jetson nano develops a board deployment visual algorithm to calculate the object type and specific coordinate value of the identified garbage, and feeds back the coordinate value to the STM32 controller, and the controller controls the stepping motor to realize the movement of the two-dimensional sliding table, so that the mechanical gripper is positioned to the garbage with the specific coordinate value, and the garbage is grabbed.
The concrete method for calculating the concrete coordinate value of the identified garbage by using jetson nano development board deployment visual algorithm comprises the following steps:
Step 1, constructing a garbage classification detection model based on YOLOv and a Normalized Attention Module (NAM) (Normalization-based Attention Module);
adopting YOLOv network model and normalized attention module NAM as main frame of garbage classification detection model, as shown in figure 4, YOLOv network model is divided into 4 parts, respectively extracting network, feature fusion network, DBL convolution layer and loss function for main feature, embedding NAM module at the end of each network block in YOLOv network, making the network lighter and improving detection efficiency;
NAM is an efficient and lightweight attention module, which directly uses scaling factors of batch normalization (Batch Normalization, BN) to calculate attention weights without requiring additional calculations and parameters such as full connection, convolution, etc., and further suppresses insignificant features by adding regularization terms, as compared to conventional attention modules. NAM adopts channel attention module (CBAM) and Space Attention Module (SAM) to redesign channel attention and space attention sub-module, in CBAM, BN is used to measure the importance of pixels, namely pixel normalization, as shown in formula (1), scaling factors in BN reflect the change of each channel and also indicate the importance of the channel, scaling factors, namely variances in BN, the larger the variances indicate the more varied the channel, the more abundant the information contained in the channel, the larger the importance, and the channels with little change, the single information and small importance.
Where μ B and σ B are the mean and standard deviation, respectively, of small lot B, and γ and β are trainable affine transformation parameters (scale and displacement).
The CBAM-channel attention sub-module is shown in formula (2) and fig. 5, where M c represents the output characteristics, sigmoid represents the activation function, γ is the scale factor of each channel, and the weight W γ=γi/∑j=0γj,F1 represents the input characteristics. If the same normalization method is used for each pixel in space, the weight of the spatial attention, i.e. pixel normalization, can be obtained;
The SAM spatial attention sub-module is shown in equation (3) and fig. 6, M s represents the output feature, sigmoid represents the activation function, ρ is the scale factor, weight W ρ=ρi/∑j=0ρj,F2 represents the input feature,
Mc=sigmoid(Wγ(BN(F1))) (2)
Ms=sigmoid(Wρ(BNs(F2)) (3)
The NAM attention module adds a regularization term to its loss function, shown in equation (4), to suppress unimportant features.
Loss=∑(x,y)l(f(x,W),y)+p∑g(γ)+p∑g(ρ) (4)
In the above formula, x is input, y is output, W is the network weight, f (·) is the input feature, l (·) is the loss function, g (·) is the l 1 norm penalty function, and p is the penalty parameter for balancing g (γ) and g (ρ).
The invention combines YOLOv with NAM module to obtain improved YOLOv network model as shown in figure 7 as garbage classification detection model;
Step2, counting common household garbage types and constructing a training data set training garbage classification detection model;
In order to realize accurate classification of multiple types of garbage, the embodiment trains a multi-classification model, can realize accurate classification of multiple types of garbage, and can identify plastic bottles, paper cups, pop cans and the like in common garbage.
Step 3, identifying and classifying the garbage put on the detection platform by using a trained garbage classification detection model, classifying the common garbage into four types of recyclable garbage, kitchen garbage, harmful garbage and other garbage in the detection process, calculating the coordinate position of the garbage put on the detection platform, and marking the coordinate position as shown in fig. 8;
step 4, returning to the relative coordinates of the target in the detection range of the camera by utilizing the target detection characteristic of the YOLO model, obtaining depth information by measuring the distance between the camera and the detection platform, mapping the target coordinates obtained by the camera to the coordinates of the detection target in the world coordinate system by vector conversion, and converting the coordinates of the detection target in the world coordinate system into the target coordinates of the two-dimensional slipway by simple proportional operation because the two-dimensional slipway has no influence of three-dimensional parameters, wherein the accurate grabbing of the target garbage can be realized by the algorithm as shown in fig. 9;
in this embodiment, the upper left corner of the detection platform is taken as the origin of the coordinate system, the detect. Py file is found in the algorithm code, and the plot_one_box is opened and found, and the code is as follows:
then, ctr+mouse clicks, enters into general. Py, and automatically locates to the plot_one_box function, and modifies the function into
And outputting the target coordinate information to obtain the coordinate position of the garbage to be grabbed.
It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will understand that they can make modifications to the technical solutions described in the above-mentioned embodiments or make equivalent substitutions of some or all of the technical features, without departing from the essence of the corresponding technical solutions from the scope of the invention defined by the claims.