CN115937765A

CN115937765A - Image identification method, AGV material sorting method and system

Info

Publication number: CN115937765A
Application number: CN202211491394.6A
Authority: CN
Inventors: 马菲飞; 单晓杭
Original assignee: Zhejiang University of Technology ZJUT
Current assignee: Zhejiang University of Technology ZJUT
Priority date: 2022-11-25
Filing date: 2022-11-25
Publication date: 2023-04-07

Abstract

The invention belongs to the field of AGV, and particularly relates to an image identification method, an AGV material sorting method and an AGV material sorting system, which comprise the following steps: s1, collecting a target image to form a self-built data set; s2, segmenting the image, and extracting a data set with material characteristics and tray information from the image; s3, performing annotation enhancement on the image; s4, constructing an improved YOLO-v5 model; s5, training an improved YOLO-v5 model to obtain an optimal training model; and S6, classifying and identifying the materials by the trained YOLO-v5 model, judging the tray load, and outputting the types of the materials and the load condition of the tray. According to the invention, the MobileNet-v3 is used as a backsbone part in the improved YOLO-v5 model, so that the problems of large quantity of backsbone structure parameters, low detection speed and the like in the original YOLO-v5 model are better solved.

Description

Image recognition method, AGV material sorting method and system

Technical Field

The invention relates to the field of AGV, in particular to an image identification method, an AGV material sorting method and an AGV material sorting system.

Background

AGVs are an abbreviation of Automated Guided Vehicle, that is, "Automated Guided vehicles," and refer to vehicles equipped with electromagnetic or optical automatic guidance devices, which can travel along a predetermined guidance path, and have safety protection and various transfer functions.

AGV possesses automatic handling and high-efficient convenient advantage, can effectively solve the transportation process of enterprise product in the production process and consume a large amount of manpower resources, consequently has obtained a large amount of applications in trades such as automobile industry, household electrical appliances manufacturing, tobacco, commodity circulation, 3C electronics, medicine, food. Along with the high-speed operation of factory production assembly line, the user has provided higher requirement to AGV system's real-time running condition and security, and AGV management monitoring dispatch system can carry out real-time state monitoring and assignment and receipt of task information to AGV, is one of the important core technology of AGV system. With the advent of industrialization 4.0, the era of smart factories will be gradually opened, and the requirements for perfect integration of intelligent devices and information technologies and high manufacturing flexibility are increasingly enhanced.

For multiple AGV management, an AGV scheduling system is required, wherein the AGV scheduling system comprises task path planning, task allocation, traffic control, vehicle management, interface control with an ERP (enterprise resource planning) and PDM (product data management) system of a client production enterprise, and the like. Although the conventional central dispatching system of the AGV can complete basic tasks dispatching, traffic control, vehicle state monitoring and other works, the flexibility degree of article handling is not high, and the dependence degree on a fixed route is too high, for example, when a handling instruction is issued, a central management system of an AGV cluster based on Web is disclosed in chinese patent application No. CN106856493, a handling start position and various parameters still need to be specified, and the AGV cannot carry automatically in some occasions; under the specific condition that materials need to be classified and carried, the common article classification method mostly depends on path planning to perform fixed partition classification, and a plurality of scenes needing human interference still exist; the conventional AGV still has the defects of cluster management, when the AGV has a transport fault, an AGV production supplier and a user cannot realize the on-site monitoring and on-line diagnosis functions, so that errors are difficult to diagnose. ,

disclosure of Invention

In order to overcome the defects in the prior art, the invention provides an image identification method, an AGV material sorting method and an AGV material sorting system.

The invention provides an image recognition method, which comprises the following steps:

s1, collecting a target image to form a self-built data set;

s2, segmenting the image, and extracting a data set with material characteristics and tray information from the image;

s3, performing annotation enhancement on the image, and dividing the data set into a training set, a verification set and a test set;

s4, constructing an improved YOLO-v5 model;

s5, training an improved YOLO-v5 model to obtain an optimal training model;

and S6, classifying and identifying the materials by the trained YOLO-v5 model, judging the tray load, and outputting the types of the materials and the load condition of the tray.

Further, the S3 includes:

labeling the article type and position information in the image by adopting Labelimg software, and ensuring that the labeling information corresponds to the labeled picture;

and performing data enhancement on the marked image, wherein the adopted method comprises the steps of clockwise rotating the image for a plurality of times by a plurality of angles, then randomly adjusting the brightness and the chromaticity of each image, increasing Gaussian noise, randomly superposing the images, and generating a corresponding marking file while enhancing the image.

Further, the S4 includes:

s4-1, adopting an improved MobileNet-v3 network structure as a backhaul part in an improved YOLO-v5 model, wherein the improved MobileNet-v3 network structure adopts a pixel-level and channel-level attention mechanism CANet to replace an SE attention mechanism in a MobileNet-v3 basic network Bnic; the network structure of the Backbone in the improved YOLO-v5 model consists of 1 3x3 convolution and 14 layers of bneck structures; firstly, reducing the characteristic size of an input image through a 3x3 convolution, and then entering a 14-layer bneck structure for characteristic extraction; the bnic structure is in a form of an inverted residual error structure, firstly, 1x1 point-by-point convolution is used for carrying out dimension increasing operation, a characteristic diagram channel is expanded, the characteristic quantity is enriched, then a CANet attention mechanism is selectively added in the 3x3 convolution operation, and finally, a final characteristic diagram is obtained through 1x1 convolution;

s4-2, a network structure design of FPN + PAN in the original YOLO-v5 model is used as a neck part in the improved YOLO-v5 model, a feature diagram obtained through S4-1 transmits feature information from bottom to top through an FPN layer, high-level features of low-resolution and high-semantic information and bottom-level features of high-resolution and low-semantic information are fused from bottom to top, and then positioning information is enhanced from top to bottom through PAN;

s4-3, a Transformer detection Head is used as a Head part in the improved YOLO-v5 model, the Transformer detection Head is obtained by utilizing the Transformer to improve the Head part in the original YOLO-v5 model, a characteristic diagram obtained in the S4-2 step is transmitted into the Transformer detection Head, and the detection of the small object is enhanced by utilizing the characteristic that the local information obtained by the Transformer is larger than CNN.

Further, the S5 includes:

the improved YOLO-v5 model is trained and learned by utilizing the training set, the weight parameters are updated, the optimal weight parameters of the training are tested by utilizing the test set, the identification accuracy and the real-time performance of the optimal weight parameters meet the VGA trolley identification standard, and the optimal training model is finally obtained.

The invention also provides an AGV material sorting method, which comprises the following steps:

step 1, entering a client through a web page login service module, starting a current working mode in the client, starting a digital camera to work at the moment, and transmitting the globally shot tray load condition to a background server through a local area network generated by a wireless AP (access point);

step 2, converting the video frame into an image by the background server, transmitting the converted image data into an image recognition module, wherein the image recognition module comprises an improved YOLO-v5 model as claimed in any one of claims 1 to 4, performing recognition and judgment by the image recognition module, and issuing a carrying task to an AGV with an industrial camera by the task management module if a tray is recognized to be loaded;

step 3, after the AGV with the industrial camera receives the carrying task, calculating a route of the current AGV position to the carrying area through a path planning module, and when the AGV drives to the carrying area according to the calculated route, acquiring a material image through the industrial camera and transmitting scanned material video information to a background server;

and 4, converting the acquired real-time video information stream into image information by the background server, transmitting the image information into the image identification module, identifying and judging by the image identification module, outputting the material type, calling the path planning module again, calculating an area where the current material type is stored, planning a path for the AGV, and finishing the carrying work by the AGV according to the planned path.

The invention further provides an AGV material sorting system which is used for realizing the AGV material sorting method and comprises a client, a background server, a wireless AP, a digital camera and an AGV, wherein the client, the background server and the wireless AP are sequentially in communication connection, the wireless AP is respectively in communication connection with the digital camera and the AGV, and the AGV is provided with an industrial camera capable of automatically rotating.

Further, the background server comprises a web webpage login service module, an image identification module, a path planning module, a task management module and a state monitoring module; the web page login service module is used for providing a login control interface based on a browser end, and can be used for online control at a PC end and field control at a mobile end; the image recognition module is used for classifying and recognizing materials, judging tray loads, and outputting the types of the materials and the load conditions of the trays; the task management module is used for issuing tasks and managing the tasks for the AGV; the path planning module is used for managing and controlling an AGV running path and a traveling track; and the state monitoring module is used for monitoring the running state of the AGV.

The invention has the beneficial effects that:

1. according to the conventional AGV sorting system and sorting method, a work starting point still needs to be set manually, and the automation degree of the AGV work is not high, but an image recognition module is adopted, the load condition of a tray is monitored through a digital camera, so that a global monitoring mode is realized, the cost of manpower and material resources is greatly saved, the digital camera can transmit the load condition of the tray to the image recognition module of a background server for judgment only by starting the work mode, and when the tray is loaded to a certain degree, the background server sends an instruction to the AGV through a task management module, so that unattended logistics intelligence is realized;

2. according to the invention, mobileNet-v3 is used as a backhaul part in an improved YOLO-v5 model, so that the problems of large structural parameter quantity, low detection speed and the like of the backhaul in the original YOLO-v5 model are better solved, and especially at a mobile end and an embedded end, the complex model in the original YOLO-v5 is difficult to utilize;

3. the SE part of the lightweight network MobileNet-v3 is replaced by a CANet attention mechanism, so that each bneck network structure in the MobileNet-v3 uses a pixel-level and channel-level attention mechanism, the information extraction of image characteristics is enhanced, and the AGV can be better ensured to keep higher material identification rate even in some factories with poorer environments;

4. the sensitivity to small object detection was increased using Transformer as the Head part in the modified YOLO-v5 model.

Drawings

FIG. 1 is a schematic diagram of an AGV material sorting system according to the present invention;

FIG. 2 is a schematic diagram of an AGV structure of an AGV material sorting system according to the present invention;

FIG. 3 is a schematic diagram of an AGV partial structure of an AGV material sorting system according to the present invention;

FIG. 4 is a flowchart of an AGV material sorting method according to the present invention;

FIG. 5 is a flow chart of an image recognition method of the present invention;

FIG. 6 is a diagram of the structure of the Bneck network in Backbone with improved YOLO-v 5;

FIG. 7 is a diagram of a CANet neural network;

FIG. 8 is a diagram of a neural network for the improved YOLO-v 5.

FIG. 9 is a diagram of the structure of the Transformer network in Neck modified with YOLO-v 5.

In the figure, 1-client, 2-background server, 3-wireless AP, 4-digital camera, 5-material, 6-AGV, 7-U type fixing frame, 8-bearing support, 9-industrial camera and 10-servo motor.

Detailed Description

The technical solutions of the present invention are further described below with reference to the drawings and the detailed description of the embodiments, and it is obvious that the described embodiments are only a part of the embodiments of the invention, but do not serve as bases for limiting the invention.

Referring to fig. 1-3, an AGV material sorting system includes a client 1, a background server 2, a wireless AP3, a digital camera 4 and an AGV6, the client, the background server and the wireless AP are in communication connection in sequence, the wireless AP is in communication connection with the digital camera and the AGV respectively, and the AGV is provided with an industrial camera 9 capable of automatically rotating.

The background server 2 comprises a web webpage login service module, an image identification module, a path planning module, a task management module and a state monitoring module; the web page login service module is used for providing a login control interface based on a browser end, and can be used for online control at a PC end and field control at a mobile end; the image recognition module is used for classifying and recognizing materials, judging tray loads, and outputting the types of the materials and the load conditions of the trays; the task management module is used for issuing tasks and managing the tasks for the AGV; the path planning module is used for managing and controlling an AGV running path and a traveling track; and the state monitoring module is used for monitoring the running state of the AGV.

The client 1 is divided into a working mode and an observer mode altogether, when the client 1 is set to be the working mode, the digital camera 4 starts to monitor the loading condition of the tray, and transmits the video data into the background server 2, the image identification module in the background server 2 for judging whether the carried tray is fully loaded judges whether the carried tray needs to be carried, if so, a carrying instruction is given through the task management module in the background server, the AGV6 with the industrial camera receives the carrying instruction through the wireless AP3 and drives to a carrying area through the path planning module, then the industrial camera 9 transmits the video data of the materials to be carried into the image identification module based on the improved YOLO-v5 for identification and classification, the basic type of the carried materials is obtained, the path planning module is called again to calculate the specific path of the carried materials, and the carrying work is completed.

AGV6 with industrial camera 9 is equipped with U type mount 7, servo motor 10, industrial camera 9, bearing retainer 8 by the AGV. U type mount 7 is connected with AGV6 afterbody for fixed industry camera 9. The industrial camera is placed at the inner side of the U-shaped fixing frame 7 to be parallel and level with the AGV6 at an included angle of 90 degrees, and the servo motor 10 is connected with the industrial camera 9 through a bearing in a fixing arm of the bearing support 8, so that the industrial camera can upwards rotate at an included angle of 45 degrees, and sufficient sight lines are provided when large-scale carrying materials are scanned. The industrial camera 9 transmits scanned material information to an image classifier image in the background server 2, if the current material type exists, next carrying work is carried out, and if the current material type does not exist, the servo motor 10 is rotated to adjust the angle upwards until the material type is obtained.

Referring to fig. 4, a method for sorting AGV materials includes the following steps:

step 1, entering a client 1 through a web page login service module, starting a current working mode in the client 1, starting a digital camera 4 to work at the moment, and transmitting the globally shot tray load condition to a background server 2 through a local area network generated by a wireless AP 3.

And 2, the background server 2 acquires a real-time video information stream of the video acquisition part, converts video frames into images by using opencv, and sets interval frames to convert the images once every 60 frames. And transmitting the converted image data into the image identification module for identification and judgment, and issuing a carrying task to the AGV6 with the industrial camera through the task management module if the tray is identified to be loaded.

And 3, after the AGV6 with the industrial camera receives the carrying task, calculating a route of the current position of the AGV6 to the carrying area through a path planning module, and when the AGV6 runs to the carrying area, starting to adjust the angle of the industrial camera 9 by the servo motor 10, and transmitting the scanned material video information to the background server 2.

And 4, converting the real-time video information stream of the acquired video acquisition part into image information by the background server 2, transmitting the image information into the image identification module, outputting the material type, and calling the path planning module again to calculate the area where the current material type needs to be stored to finish the carrying work.

Specifically, when the AGV6 with the industrial camera fails, the monitoring video in the failure time period can be called through the state monitoring module in the background server 2, the failure reason can be restored, and the failure problem can be better solved.

Referring to fig. 5 to 9, the image recognition module adopts the following image recognition method, which includes the following steps:

s1, acquiring a target image, wherein a self-built data set adopted by the invention comprises 5220 pictures, and the pictures comprise material types and trays which are common in logistics transportation. The self-built data set adopted by the invention converts the real-time video information stream collected by the background server into the image through the opencv framework.

S2, image segmentation, wherein the sizes of the images converted by the opencv framework are different, the images are directly placed into a YOLO-v5 model for detection, and the images can be adaptively scaled to 640x640. Large scale image scaling can result in the loss of small sized features. Therefore, in order to ensure the accuracy of the data set to be reliable, the image needs to be cut into 640x640, and then the data set with the material characteristics and the tray information is extracted from the image.

S3, image annotation enhancement, namely dividing the data set into a training set, a verification set and a test set, and comprising the following steps:

in the material classification, in order to reduce the influence of the detection environment on the result, the materials are classified into common boxes, fragile boxes, foods, fresh foods and the like. And labeling the article type and position information in the image by adopting Labelimg software, wherein the labeled information is ensured to correspond to the labeled picture.

Data enhancement of the annotated image is required. The adopted method comprises the steps of randomly adjusting the brightness and the chromaticity, rotating the image by 90 degrees clockwise for three times, then randomly adjusting the brightness and the chromaticity of each image, increasing Gaussian noise, randomly overlapping the images and the like. The image is enhanced and a corresponding annotation file is generated. And finally, the proportion of the prepared data set, the training set, the verification set and the test set is 8:1:1.

s4, constructing an improved YOLO-v5 model, as shown in FIG. 7, including:

s4-1, as shown in FIG. 8, modified MobileNet-v3 was used as the Backbone moiety in the modified YOLO-v5 model. According to the improved MobileNet-v3 network structure, a pixel-level attention mechanism CANet and a channel-level attention mechanism CANet are adopted to replace an SE attention mechanism in a MobileNet-v3 basic network Bnegk, so that extraction of more information in an image is further enhanced, and the material identification capability of an AGV in a severe environment is enhanced. The network structure of Backbone in the improved YOLO-v5 model consists of 1 convolution with 3x3 and 14 bneck structures. The input image is firstly subjected to 3x3 convolution to reduce the feature size, and then enters a 14-layer bneck structure for feature extraction. The bneck structure is combined with the characteristics of the conventional MobileNet structure as shown in fig. 5, the overall structure still adopts an inverted residual structure, firstly, 1x1 point-by-point convolution is used for dimensionality increasing operation, a feature diagram channel is expanded, the feature quantity is enriched, then, a CANet attention mechanism is selectively added in 3x3 convolution operation, and finally, 1x1 convolution is carried out to obtain a final feature diagram.

S4-2, according to the graph shown in FIG. 8, the neck part in the improved YOLO-v5 model is constructed and the network structure design of FPN + PAN in the original YOLO-v5 model is adopted. The feature map obtained through s4-1 conveys feature information from bottom to top as shown in fig. 7 by the FPN layer, and the high-level features of the low-resolution and high-semantic information and the bottom-level features of the high-resolution and low-semantic information are fused from bottom to top, so that the features under different scales have rich semantic information. And then the positioning information is strengthened from top to bottom through the PAN.

S4-3, according to the graph shown in FIG. 8, a new detection Head is selected for a Head part in the improved YOLO-v5 model, a Transformer is used for improving the original Head part, the graph is shown in FIG. 9, the feature graph obtained in the S4-2 is transmitted into the Transformer detection Head, and the detection of small objects is enhanced by the characteristic that the local information obtained by the Transformer is larger than CNN.

The transform detection head is composed of two sublayers, namely an MHD layer and an MLP layer, residual connection exists between the two sublayers, and LayerNorm and Dropout layers between the two sublayers contribute to better convergence of a network and prevent the network from being over-fitted. Wherein the MHD layer can help to obtain semantics while focusing on the pixel at present; the MLP layer is used to slow down the convergence speed.

S5, training the model, namely training and learning the improved YOLO-v5 model by utilizing a training set of common material classification and tray load, updating the weight parameters, testing the trained optimal weight parameters by using the testing set, ensuring that the identification accuracy and real-time performance of the model meet VGA trolley identification standards, and finally obtaining the optimal training model.

The above embodiments are only preferred embodiments of the present invention, and are not intended to limit the technical solutions of the present invention, so long as the technical solutions can be realized on the basis of the above embodiments without creative efforts, which should be considered to fall within the protection scope of the patent of the present invention.

Claims

1. An image recognition method, comprising:

s1, collecting a target image to form a self-built data set;

s3, carrying out image annotation enhancement, and dividing a data set into a training set, a verification set and a test set;

s4, constructing an improved YOLO-v5 model;

s5, training an improved YOLO-v5 model to obtain an optimal training model;

s6, classifying and identifying the materials by the trained YOLO-v5 model, judging the tray load, and outputting the types of the materials and the load condition of the tray.

2. The image recognition method according to claim 1, wherein the S3 includes:

and carrying out data enhancement on the marked image, wherein the adopted method comprises the steps of clockwise rotating the image for a plurality of times by a plurality of angles, then randomly adjusting the brightness and the chromaticity of each image, increasing Gaussian noise, randomly superposing the images, and generating a corresponding marking file while enhancing the image.

3. The image recognition method according to claim 1, wherein the S4 includes:

s4-1, adopting an improved MobileNet-v3 network structure as a backhaul part in an improved YOLO-v5 model, wherein the improved MobileNet-v3 network structure adopts a pixel-level and channel-level attention mechanism CANet to replace an SE attention mechanism in a MobileNet-v3 basic network Bnic; the network structure of the Backbone in the improved YOLO-v5 model consists of 1 3x3 convolution and 14 layers of bneck structures; firstly, reducing the characteristic size of an input image through a 3x3 convolution, and then entering a 14-layer bneck structure for characteristic extraction; the bnic structure is in an inverted residual error structure form, firstly, 1x1 point-by-point convolution is used for carrying out dimensionality increasing operation, a characteristic diagram channel is expanded, the characteristic quantity is enriched, then, a CANet attention mechanism is selectively added in 3x3 convolution operation, and finally, a final characteristic diagram is obtained through 1x1 convolution;

s4-3, a Transformer detection Head is used as a Head part in the improved YOLO-v5 model, the Transformer detection Head is obtained by improving the Head part in the original YOLO-v5 model through a Transformer, a characteristic diagram obtained in the S4-2 is transmitted into the Transformer detection Head, and the detection of the small object is enhanced by utilizing the characteristic that local information obtained by the Transformer is larger than CNN.

4. The image recognition method according to claim 1, wherein the S5 includes:

5. An AGV material sorting method is characterized by comprising the following steps:

step 1, entering a client through a web page login service module, starting a current working mode in the client, starting a digital camera to work at the moment, and transmitting the globally shot tray load condition to a background server through a local area network generated by a wireless AP;

step 3, after the AGV with the industrial camera receives the carrying task, calculating a route from the current position of the AGV to a carrying area through a path planning module, and when the AGV drives to the carrying area according to the calculated route, acquiring a material image by the industrial camera and transmitting scanned material video information to a background server;

6. An AGV material sorting system is used for achieving the AGV material sorting method according to claim 5, and is characterized by comprising a client, a background server, wireless APs, a digital camera and an AGV, wherein the client, the background server and the wireless APs are sequentially in communication connection, the wireless APs are respectively in communication connection with the digital camera and the AGV, and the AGV is provided with an industrial camera capable of automatically rotating.

7. The AGV material sorting system of claim 6, wherein said backend server comprises a web page login service module, an image recognition module, a path planning module, a task management module and a status monitoring module; the web page login service module is used for providing a login control interface based on a browser end, and can be used for online control at a PC end and field control at a mobile end; the image recognition module is used for classifying and recognizing materials, judging tray loads, and outputting the types of the materials and the load conditions of the trays; the task management module is used for issuing tasks and managing the tasks for the AGV; the path planning module is used for managing and controlling an AGV running path and a traveling track; and the state monitoring module is used for monitoring the running state of the AGV.